上一个帖子猜测是tokio的容器问题导致了内存无法即时释放,后来上tokio的社区提了issue,但没有代码和日志似乎讨论不下去了,只能整理好代码发出来了。(之前的测试代码都是unused和单字母变量加一堆乱七八糟的注释,所以没能第一时间发出来)
看完今天这篇文章后,有问题大伙可以讨论,如果觉得这个议题有意义,可以到issue下支持tokio社区去增加官方刷新内存的方法,当然大佬们觉得有时间的话,也可以在这个议题上为tokio贡献代码。
issue地址:Tokio Might Need a Safe Method to Refresh Runtime #6760
- 结论
整理了代码以后,我主要测试了4种状态下内存情况,每个状态启动4-5秒后会进行http2的百万并发测试,图表里反映的是前两轮的测试状态。
在图表之中,分别是左上角的默认分配器下不刷新运行时,右上角的mimalloc不刷新运行时,以及左下角的默认分配器下刷新运行时,以及右下角的mimalloc刷新运行时。
图表的横坐标是服务器启动时间,单位为秒。
图表左边的纵坐标是内存大小,作用于inner_mem(蓝色面积)和outer_mem(黄色面积)的数据,分别代表内部追踪的进程内存大小和外部监控的内存大小,单位是MB,右边的纵坐标是句柄连接数和并发任务个数统计(绿色折线),最大连接数是100万,最大任务数会是1,000,008。
内部的内存追踪是通过修改全局内存分配器实现的,外部监控进程内存是通过第三方库sysinfo实现的,外部监控的进程内存数值不稳定,但基本与linux的top命令或者图形界面的任务管理器里的数值一致。
通过以上这四个折现图可以看出,当进行百万并发测试时,运行时的实时任务增加时,内存会急剧上涨,最高时到达2500MB左右。
内部监控的内存,总是能够在任务减少的时候自动清除,说明代码确实不存在内存泄漏问题。
但如果使用默认分配器,会发现,即使时任务减少时,外部监控的内存甚至还会进一步增长,只有连接数清零时,内存才会释放,但释放过后也还会有2000MB左右。
如果使用mi_malloc分配器,外部监控的内存会随着协程任务结束而释放,但内存也不会减少到一个很低的数字,而是在600MB左右就不会再减少了。
在并发结束后,如果刷新运行时(大约在30秒和65秒的时候),可以发现外部监控内存可以回缩到极低的水平。
使用默认分配器时,外部监控的内存大概回缩一两次就不会再回缩了,不过最终内存会在10-100M之间。
使用mi_malloc分配器,外部监控的内存能逐步回缩到15M左右,非常稳定。
由以上测试结果可以看出,刷新tokio的运行时,对外部监控内存的释放确实是有意义的。
- 客户端
客户端就是一个reqwest实现的高并发客户端,cpu是8线程所以初始化了8个http2客户端,拷贝代码测试的时候可以自行调整。能够设定多轮百万并发,默认每轮中途休息10秒(方便观察服务端的内存回缩),我这边客户端的工作内存是7-8G,静止内存是4G左右。
rust版本: rustc 1.82.0-nightly (7120fdac7 2024-07-25)
cargo.toml:
[dependencies]
reqwest = "0.12.5"
tokio = { version = "1.39.2", features = ["full"] }
src/main.rs:
use std::time::{Duration, Instant};
/// 发起请求时获取地址,如果有多个目标服务器,可以在这里写地址轮询
fn get_url(round: usize) -> &'static str {
"http://0.0.0.0:3000?server=axum&expire=20000"
}
/// 循环轮次
const TOTAL_ROUNDS: usize = 300;
/// 每个轮次的请求数
const TOTAL_REQUESTS: usize = 1_000_000;
/// 每个轮次的时间间隔,单位为秒
const WAITING_SECS: u64 = 10;
/// 客户端数
const NUM_CLIENTS: usize = 8;
/// 间隔时的读秒输出
async fn countdown(seconds: u64) {
let mut interval = tokio::time::interval(Duration::from_secs(1));
for i in 0..=seconds {
interval.tick().await;
print!("\rwaiting for ({}/{}) seconds...", i, seconds);
use std::io::Write;
std::io::stdout().flush().unwrap();
}
println!("\r{}\r", " ".repeat(30));
}
/// 计算响应的百分比时间数据
fn calculate_percentiles(durations: Vec<Duration>) -> [Duration; 5] {
let mut sorted_durations = durations;
sorted_durations.sort_by_key(|d| d.as_millis());
let total_requests = sorted_durations.len();
[
sorted_durations[total_requests / 2],
sorted_durations[total_requests * 9 / 10],
sorted_durations[total_requests * 99 / 100],
sorted_durations[total_requests * 999 / 1000],
sorted_durations.last().cloned().unwrap_or_default(),
]
}
#[tokio::main]
async fn main() {
let start = Instant::now();
let clients: Vec<reqwest::Client> = (0..NUM_CLIENTS)
.map(|_| {
reqwest::ClientBuilder::new()
// .danger_accept_invalid_certs(true) // 测试自签https用的
.pool_idle_timeout(Duration::from_secs(1))
.http2_prior_knowledge() // http2的客户端标识
.build()
.unwrap()
})
.collect();
println!("build-clients-time: {:.3?}", start.elapsed());
for round in 1..=TOTAL_ROUNDS {
multi_requests(round, &clients).await;
countdown(WAITING_SECS).await;
}
}
/// 每轮的并发请求函数
async fn multi_requests(round: usize, clients: &Vec<reqwest::Client>) {
let runtime = tokio::runtime::Runtime::new().unwrap();
let mut results = std::collections::HashMap::new();
let mut successful_times = Vec::new();
let mut successful_count = 0u32;
let clients_len = clients.len();
let start = Instant::now();
let mut tasks = Vec::with_capacity(TOTAL_REQUESTS);
for i in 0..TOTAL_REQUESTS {
let client = clients[i % clients_len].clone();
tasks.push(runtime.spawn(async move {
let url = get_url(round);
let request_start = Instant::now();
let response = client
.get(url)
.timeout(Duration::from_secs(60))
.send()
.await;
let result = match response {
Ok(r) => r.text().await,
Err(e) => Err(e),
};
(result, request_start.elapsed())
}));
}
println!("{round} - prepared-time: {:.3?}", start.elapsed());
// 统计的时候把所有响应和错误类型都转成String来统计了
for task in tasks {
let (result, elapsed) = task.await.unwrap();
if result.is_ok() {
successful_times.push(elapsed);
successful_count += 1;
}
let result = result.unwrap_or_else(|e| e.to_string());
*results.entry(result).or_insert(0) += 1;
}
println!("{round} - completed-time: {:.3?}", start.elapsed());
// 存在成功响应时,统计输出一些具体的数据
if successful_count > 0 {
println!(
"{round} - successful-count: {successful_count}, average-time: {:.3?}",
successful_times.iter().sum::<Duration>() / (successful_count as u32)
);
let percentiles = calculate_percentiles(successful_times);
println!(
"{round} - percentiles: {{ 50%: {:.3?}, 90%: {:.3?}, 99%: {:.3?}, 99.9%: {:.3?}, 100%: {:.3?} }}",
percentiles[0], percentiles[1], percentiles[2], percentiles[3], percentiles[4]
);
}
// 响应成功率计算
println!(
"{round} - results-len: {}, success-rate: {:.2}%",
results.len(),
100. * successful_count as f64 / TOTAL_REQUESTS as f64
);
// 响应结果太多时就不输出了,通常在测试http1.1的时候不会输出结果。
if results.len() <= 100 {
println!("{round} - results: {:#?}", results);
}
runtime.shutdown_background();
}
典型的输出数据如下:
build-clients-time: 61.264ms
1 - prepared-time: 2.401s
1 - completed-time: 26.181s
1 - successful-count: 1000000, average-time: 18.341s
1 - percentiles: {50%: 18.743s, 90%: 21.764s, 99%: 22.663s, 99.9%: 23.372s, 100%: 24.031s}
1 - results-len: 8, success-rate: 100.00%
1 - results: {
"axum(20027) HTTP/2.0 192.168.0.123:11582": 125000,
"axum(20027) HTTP/2.0 192.168.0.123:11579": 125000,
"axum(20027) HTTP/2.0 192.168.0.123:11581": 125000,
"axum(20027) HTTP/2.0 192.168.0.123:11578": 125000,
"axum(20027) HTTP/2.0 192.168.0.123:11583": 125000,
"axum(20027) HTTP/2.0 192.168.0.123:11584": 125000,
"axum(20027) HTTP/2.0 192.168.0.123:11580": 125000,
"axum(20027) HTTP/2.0 192.168.0.123:11585": 125000,
}
2 - prepared-time: 2.393s
2 - completed-time: 25.683s
2 - successful-count: 1000000, average-time: 18.667s
2 - percentiles: {50%: 18.503s, 90%: 22.145s, 99%: 22.761s, 99.9%: 23.732s, 100%: 24.385s}
2 - results-len: 8, success-rate: 100.00%
2 - results: {
"axum(20027) HTTP/2.0 192.168.0.123:11591": 125000,
"axum(20027) HTTP/2.0 192.168.0.123:11586": 125000,
"axum(20027) HTTP/2.0 192.168.0.123:11587": 125000,
"axum(20027) HTTP/2.0 192.168.0.123:11593": 125000,
"axum(20027) HTTP/2.0 192.168.0.123:11592": 125000,
"axum(20027) HTTP/2.0 192.168.0.123:11590": 125000,
"axum(20027) HTTP/2.0 192.168.0.123:11589": 125000,
"axum(20027) HTTP/2.0 192.168.0.123:11588": 125000,
}
waiting for (3/10) seconds...
57 - prepared-time: 2.181s
57 - completed-time: 67.254s
57 - successful-count: 606228, average-time: 17.133s
57 - percentiles: { 50%: 16.718s, 90%: 18.246s, 99%: 19.710s, 99.9%: 59.925s, 100%: 60.001s }
57 - results-len: 10, success-rate: 60.62%
57 - results: {
"axum(13053) HTTP/2.0 192.168.0.123:7806": 124806,
"axum(13053) HTTP/2.0 192.168.0.123:7808": 58174,
"axum(13053) HTTP/2.0 192.168.0.123:7805": 8790,
"axum(13053) HTTP/2.0 192.168.0.123:7811": 64255,
"error decoding response body": 86967,
"axum(13053) HTTP/2.0 192.168.0.123:7807": 116657,
"axum(13053) HTTP/2.0 192.168.0.123:7810": 122746,
"axum(13053) HTTP/2.0 192.168.0.123:7812": 107818,
"error sending request for url (http://192.168.0.127:3000/?server=axum&expire=15000)": 306805,
"axum(13053) HTTP/2.0 192.168.0.123:7809": 2982,
}
正确的响应包括:服务器类型标识,服务端进程id,http类型,以及客户端的ip地址。一共100万个请求,8个客户端,每个客户端分到12.5万。
如果在请求的过程中,特别是服务端链接已经开始释放的时候,ctrl+c
关闭服务端,就可以观测到错误输出了,一般只有sending和decoding两种错误。
和上次写文章的时候相比,改了一些地方,内存开销小了很多(之前的版本把统计放进了并发的协程里,工作内存在12G这样),响应性能区别不大。
- 服务端
服务端是axum+hyper的low-level模式,代码比较长,包括了内存监控(同时监控分配器的内部内存和外部进程的内存),全局运行时的监控和替换(这里直接用parking_lot
来加锁修改了,因为数据竞争很小,所以加锁操作对性能损耗并不明显),句柄设计为根据params
里提取expire
来实现动态休眠,同时用一个my_status
变量来跟跟踪句柄的链接状态,另外另起了一个协程来在命令行里输出这些状态。
rust在排查内存泄漏的时候,一般都是通过内部的内存分配器来监控内存状态(下述简称为内部追踪的内存为INNER_MEMORY),但运维管理时我们通常只会监控外部的内存(下述简称为外部检测的内存为OUTER_MEMORY),这两个内存肯定是不一样的,内部内存稳定通常代表没有内存泄漏,外部内存远大于内部内存,通常代表进程的内存没有即时归还给系统。
mi_malloc目前主要适配于linux,其他系统可能需要注释掉和mi_malloc相关的代码,但不影响对进程内存是否释放的观察。
可以通过切换注释来切换内存分配器以及是否刷新tokio的运行时,切换注释后需要重新进行增量编译。
static REAL_ALLOCATOR: (std::alloc::System, &str, u64) = (std::alloc::System, "DefaultAlloc", 20);
// static REAL_ALLOCATOR: (mimalloc::MiMalloc, &str, u64) = (mimalloc::MiMalloc, "MiMalloc", 20);
// const AUTO_REFRESH: bool = false;
const AUTO_REFRESH: bool = true;
rust版本: rustc 1.82.0-nightly (7120fdac7 2024-07-25)
cargo.toml:
[dependencies]
axum = { version = "0.7.5", features = ["http2"] }
tokio = { version = "1.39.2", features = ["full"] }
mimalloc = "0.1.42"
hyper = "1.4.1"
hyper-util = { version = "0.1.7", features = ["tokio", "server-auto", "http2"] }
tower = { version = "0.4.13", features = ["full"] }
sysinfo = "0.30.13"
parking_lot = "0.12.3"
src/main.rs:
use std::alloc::{GlobalAlloc, Layout};
use std::collections::HashMap;
use std::net::SocketAddr;
use std::sync::{Arc, LazyLock};
use std::sync::atomic::{AtomicUsize, Ordering};
use std::time::Duration;
use sysinfo::{Pid, System};
use tokio::runtime::{Handle, Runtime};
use hyper::body::Incoming;
use hyper_util::rt::{TokioExecutor, TokioIo};
use hyper_util::server;
use tower::Service;
use axum::extract::{ConnectInfo, Query, Request, State};
use axum::{Extension, Router, routing::get};
/// 定义全局运行时
use parking_lot::RwLock;
static GLOBAL_RUNTIME: RwLock<Option<Runtime>> = RwLock::new(None);
/// 进程id
static PID: LazyLock<Pid> = LazyLock::new(|| Pid::from_u32(std::process::id()));
/// 监视器在命令行里的输出频率,默认时1秒输出一次
const INTERVAL_SECS: u64 = 1;
/// 自定义的内存分配器的追踪器
struct TrackedAlloc;
#[global_allocator]
static ALLOC: TrackedAlloc = TrackedAlloc;
/// 内部内存追踪
static INNER_MEMORY: AtomicUsize = AtomicUsize::new(0);
/// 真实的分配器,第一种是默认分配器,第二种是MiMalloc,同一个代码只能用一个分配器,所以只能通过注释来选择性编译
/// 元组的三个元素分别是分配器,分配器名称,内存收缩时的阈值(即内存在这个值(单位:MB)之上时会刷新tokio的运行时)
static REAL_ALLOCATOR: (std::alloc::System, &str, u64) = (std::alloc::System, "DefaultAlloc", 20);
// static REAL_ALLOCATOR: (mimalloc::MiMalloc, &str, u64) = (mimalloc::MiMalloc, "MiMalloc", 20);
/// 是否自动刷新tokio的运行时
// const AUTO_REFRESH: bool = false;
const AUTO_REFRESH: bool = true;
/// 内部追踪内存的具体实现
unsafe impl GlobalAlloc for TrackedAlloc {
unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
let ret = REAL_ALLOCATOR.0.alloc(layout);
if !ret.is_null() {
INNER_MEMORY.fetch_add(layout.size(), Ordering::Relaxed);
}
ret
}
unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) {
REAL_ALLOCATOR.0.dealloc(ptr, layout);
INNER_MEMORY.fetch_sub(layout.size(), Ordering::Relaxed);
}
unsafe fn alloc_zeroed(&self, layout: Layout) -> *mut u8 {
let ret = REAL_ALLOCATOR.0.alloc_zeroed(layout);
if !ret.is_null() {
INNER_MEMORY.fetch_add(layout.size(), Ordering::Relaxed);
}
ret
}
unsafe fn realloc(&self, ptr: *mut u8, layout: Layout, new_size: usize) -> *mut u8 {
let ret = REAL_ALLOCATOR.0.realloc(ptr, layout, new_size);
if !ret.is_null() {
INNER_MEMORY.fetch_add(new_size.wrapping_sub(layout.size()), Ordering::Relaxed);
}
ret
}
}
/// 句柄状态的追踪
#[derive(Debug, Default)]
struct MyState {
counter: AtomicUsize,
connection: AtomicUsize,
max_connection: AtomicUsize,
}
#[tokio::main]
async fn main() {
// 初始化或替换全局运行时,通过写锁来设置
GLOBAL_RUNTIME.write().replace(Runtime::new().unwrap());
println!("axum-demo pid is {}", *PID);
// 监听端口
let listener = tokio::net::TcpListener::bind("0.0.0.0:3000").await.unwrap();
println!("listening on {}", listener.local_addr().unwrap());
// 初始化句柄状态
let my_status: Arc<MyState> = Default::default();
// 通过main运行时来启动异步监视器
tokio::spawn(minitor(my_status.clone()));
// 创建axum服务
let app = Router::new().route("/", get(index)).with_state(my_status);
// 以low-level的形式循环建立tcp连接
while let Ok((stream, addr)) = listener.accept().await {
let app = app.clone();
// spawn时,通过读锁来获取全局运行时
GLOBAL_RUNTIME.read().as_ref().unwrap().spawn(async move {
let io = TokioIo::new(stream);
let hyper_service = hyper::service::service_fn(move |req: Request<Incoming>| {
// 将每个连接的地址信息注入到句柄里
app.clone().layer(Extension(ConnectInfo(addr))).call(req)
});
if let Err(err) = server::conn::auto::Builder::new(TokioExecutor::new())
.http2() // http2 服务设置标识
.max_concurrent_streams(1000000) // 单条http2链接的最大并发流数
.keep_alive_timeout(Duration::from_secs(1))
.serve_connection_with_upgrades(io, hyper_service)
.await
{
println!("error serving connection from {addr:?}: {err:#}");
}
});
}
}
/// 句柄,会从params里提取expire来进行休眠,同时刷新my_status的connection和计数状态
async fn index(
version: axum::http::Version,
ConnectInfo(addr): ConnectInfo<SocketAddr>,
State(my_status): State<Arc<MyState>>,
Query(params): Query<HashMap<String, String>>,
) -> String {
my_status.counter.fetch_add(1, Ordering::Relaxed);
let conn = 1 + my_status.connection.fetch_add(1, Ordering::Relaxed);
if conn > my_status.max_connection.load(Ordering::Relaxed) {
my_status.max_connection.store(conn, Ordering::Relaxed);
}
if let Some(Ok(expire)) = params.get("expire").map(|s| s.parse::<u64>()) {
tokio::time::sleep(Duration::from_millis(expire)).await;
}
my_status.connection.fetch_sub(1, Ordering::Relaxed);
format!("axum({}) {version:?} {addr:?}", *PID)
}
/// 通过sysinfo第三方库从外部获取进程内存
fn get_memory_usage() -> u64 {
let system_info = System::new_all();
let current_process = system_info.process(*PID).expect("Process not found");
current_process.memory()
}
/// 异步函数实现的监视器
async fn minitor(my_status: Arc<MyState>) {
let mut interval = tokio::time::interval(Duration::from_secs(INTERVAL_SECS));
let mut tick = 0;
loop {
interval.tick().await;
let ct = my_status.counter.load(Ordering::Relaxed);
let conn = my_status.connection.load(Ordering::Relaxed);
let max_conn = my_status.max_connection.load(Ordering::Relaxed);
let outer_memory = get_memory_usage();
let global_spawn = GLOBAL_RUNTIME
.read()
.as_ref()
.unwrap()
.handle()
.metrics()
.num_alive_tasks();
let main_spawn = Handle::current().metrics().num_alive_tasks();
// 刷新全局运行时,当全局运行时没有任务时,且外部内存超过阈值时,刷新全局运行时
let refresh_message =
if AUTO_REFRESH && global_spawn == 0 && outer_memory > REAL_ALLOCATOR.2 * 1024 * 1024 {
let start = std::time::Instant::now();
// 把旧的全局运行时替换出来
let old_global_runtime = GLOBAL_RUNTIME.write().replace(Runtime::new().unwrap());
// 在main运行时里,关闭旧的全局运行时
old_global_runtime.unwrap().shutdown_background();
format!(
", refresh runtime on {} until {}M! cost {:?}",
REAL_ALLOCATOR.1,
REAL_ALLOCATOR.2,
start.elapsed()
)
} else {
"".to_string()
};
// 输出句柄的计数器、实时连接数、最大连接数、内部内存、外部内存、主运行时正在运行的任务、全局运行时正在运行的任务、以及是否刷新的信息
println!(
"axum {tick}, ct: {ct}, conn: {conn}, max_conn: {max_conn}, inner_mem: {:.3}M, outer_mem: {:.3}M, main_rt: {}, global_rt: {}{}",
INNER_MEMORY.load(Ordering::Relaxed) as f64 / 1024. / 1024.,
outer_memory as f64 / 1024. / 1024.,
main_spawn,
global_spawn,
refresh_message,
);
tick += INTERVAL_SECS;
}
}
在整理完代码后,服务端删掉了很多路由和其他监控,最大内存只有2500MB左右了(之前的文章里是3.5G-4.5G),可以勉强得出一个实践经验,tokio::spawn
创建协程时,克隆拷贝进入协程的数据和状态越复杂,内存开销就越大,而且更加不容易释放。
这个代码中的方法并不是一个刷新运行时的最佳实践,刷新全局运行时必须要等到实时任务归零,这意味着需要在运行时之外才能刷新当前运行时。
而且实时任务归零的过程需要等待,比如等待所有长连接结束才能进入刷新过程。
即使是实时任务归零了,也不代表能够完全安全的关闭全局运行时,实时任务归零只能代表服务端正在运行的异步任务结束了,但tcp连接可能还没有完全断开,这个时候关闭全局运行时有可能会把tcp连接关闭,导致客户端tcp缓冲区的数据不一定能安全读取。
目前这个刷新方法只适合用于axum/hyper,其他一些依赖tokio的框架并不能安全使用。
我也不确定tokio刷新运行时的线程以及的内部的容器能否让系统完全回收内存,这需要熟悉tokio运行时代码的编码者进一步详细的测试,但整体替换运行时确实有助于系统回收内存。
- 日志
图表对应的服务端日志原始数据如下:
1、default-without-refresh
axum-demo pid is 2133
listening on 0.0.0.0:3000
axum 0, ct: 0, conn: 0, max_conn: 0, inner_mem: 0.258M, outer_mem: 6.125M, main_rt: 1, global_rt: 0
axum 1, ct: 0, conn: 0, max_conn: 0, inner_mem: 0.258M, outer_mem: 6.508M, main_rt: 1, global_rt: 0
axum 2, ct: 0, conn: 0, max_conn: 0, inner_mem: 0.258M, outer_mem: 6.527M, main_rt: 1, global_rt: 0
axum 3, ct: 0, conn: 0, max_conn: 0, inner_mem: 0.258M, outer_mem: 6.684M, main_rt: 1, global_rt: 0
axum 4, ct: 27760, conn: 27760, max_conn: 27760, inner_mem: 83.410M, outer_mem: 87.824M, main_rt: 1, global_rt: 31690
axum 5, ct: 237193, conn: 237193, max_conn: 237193, inner_mem: 643.367M, outer_mem: 636.449M, main_rt: 1, global_rt: 243295
axum 6, ct: 409585, conn: 409585, max_conn: 409585, inner_mem: 969.766M, outer_mem: 1003.730M, main_rt: 1, global_rt: 413277
axum 7, ct: 588048, conn: 588048, max_conn: 588048, inner_mem: 1609.142M, outer_mem: 1668.973M, main_rt: 1, global_rt: 690160
axum 8, ct: 860580, conn: 860580, max_conn: 860580, inner_mem: 2113.805M, outer_mem: 2263.973M, main_rt: 1, global_rt: 933078
axum 9, ct: 1000000, conn: 1000000, max_conn: 1000000, inner_mem: 2242.541M, outer_mem: 2445.973M, main_rt: 1, global_rt: 1000008
axum 10, ct: 1000000, conn: 1000000, max_conn: 1000000, inner_mem: 2242.541M, outer_mem: 2445.973M, main_rt: 1, global_rt: 1000008
axum 11, ct: 1000000, conn: 1000000, max_conn: 1000000, inner_mem: 2242.541M, outer_mem: 2445.973M, main_rt: 1, global_rt: 1000008
axum 12, ct: 1000000, conn: 1000000, max_conn: 1000000, inner_mem: 2242.541M, outer_mem: 2445.973M, main_rt: 1, global_rt: 1000008
axum 13, ct: 1000000, conn: 1000000, max_conn: 1000000, inner_mem: 2242.541M, outer_mem: 2445.973M, main_rt: 1, global_rt: 1000008
axum 14, ct: 1000000, conn: 1000000, max_conn: 1000000, inner_mem: 2242.541M, outer_mem: 2445.973M, main_rt: 1, global_rt: 1000008
axum 15, ct: 1000000, conn: 1000000, max_conn: 1000000, inner_mem: 2242.541M, outer_mem: 2445.973M, main_rt: 1, global_rt: 1000008
axum 16, ct: 1000000, conn: 1000000, max_conn: 1000000, inner_mem: 2242.541M, outer_mem: 2445.973M, main_rt: 1, global_rt: 1000008
axum 17, ct: 1000000, conn: 1000000, max_conn: 1000000, inner_mem: 2242.541M, outer_mem: 2445.973M, main_rt: 1, global_rt: 1000008
axum 18, ct: 1000000, conn: 1000000, max_conn: 1000000, inner_mem: 2242.541M, outer_mem: 2445.973M, main_rt: 1, global_rt: 1000008
axum 19, ct: 1000000, conn: 983001, max_conn: 1000000, inner_mem: 2221.630M, outer_mem: 2450.598M, main_rt: 1, global_rt: 982367
axum 20, ct: 1000000, conn: 795699, max_conn: 1000000, inner_mem: 1998.540M, outer_mem: 2496.223M, main_rt: 1, global_rt: 795714
axum 21, ct: 1000000, conn: 617085, max_conn: 1000000, inner_mem: 1834.297M, outer_mem: 2547.473M, main_rt: 1, global_rt: 614342
axum 22, ct: 1000000, conn: 490067, max_conn: 1000000, inner_mem: 1542.834M, outer_mem: 2557.598M, main_rt: 1, global_rt: 490082
axum 23, ct: 1000000, conn: 303630, max_conn: 1000000, inner_mem: 1334.821M, outer_mem: 2615.848M, main_rt: 1, global_rt: 303645
axum 24, ct: 1000000, conn: 138866, max_conn: 1000000, inner_mem: 1049.048M, outer_mem: 2634.598M, main_rt: 1, global_rt: 135432
axum 25, ct: 1000000, conn: 16145, max_conn: 1000000, inner_mem: 802.302M, outer_mem: 2649.473M, main_rt: 1, global_rt: 15351
axum 26, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 683.543M, outer_mem: 2638.582M, main_rt: 1, global_rt: 8
axum 27, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 683.543M, outer_mem: 2638.582M, main_rt: 1, global_rt: 8
axum 28, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 683.543M, outer_mem: 2638.582M, main_rt: 1, global_rt: 8
axum 29, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 0.258M, outer_mem: 2062.879M, main_rt: 1, global_rt: 0
axum 30, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 0.258M, outer_mem: 2062.879M, main_rt: 1, global_rt: 0
axum 31, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 0.258M, outer_mem: 2062.879M, main_rt: 1, global_rt: 0
axum 32, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 0.258M, outer_mem: 2062.879M, main_rt: 1, global_rt: 0
axum 33, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 0.258M, outer_mem: 2062.879M, main_rt: 1, global_rt: 0
axum 34, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 0.258M, outer_mem: 2062.879M, main_rt: 1, global_rt: 0
axum 35, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 0.258M, outer_mem: 2062.879M, main_rt: 1, global_rt: 0
axum 36, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 0.258M, outer_mem: 2062.879M, main_rt: 1, global_rt: 0
axum 37, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 0.258M, outer_mem: 2062.879M, main_rt: 1, global_rt: 0
axum 38, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 0.258M, outer_mem: 2062.879M, main_rt: 1, global_rt: 0
axum 39, ct: 1146947, conn: 146947, max_conn: 1000000, inner_mem: 375.161M, outer_mem: 2062.879M, main_rt: 1, global_rt: 150038
axum 40, ct: 1330594, conn: 330594, max_conn: 1000000, inner_mem: 1093.399M, outer_mem: 2174.879M, main_rt: 1, global_rt: 430367
axum 41, ct: 1646670, conn: 646670, max_conn: 1000000, inner_mem: 1597.347M, outer_mem: 2236.879M, main_rt: 1, global_rt: 652922
axum 42, ct: 1794766, conn: 794766, max_conn: 1000000, inner_mem: 2163.816M, outer_mem: 2465.754M, main_rt: 1, global_rt: 986139
axum 43, ct: 2000000, conn: 1000000, max_conn: 1000000, inner_mem: 2235.042M, outer_mem: 2532.879M, main_rt: 1, global_rt: 1000008
axum 44, ct: 2000000, conn: 1000000, max_conn: 1000000, inner_mem: 2235.042M, outer_mem: 2532.879M, main_rt: 1, global_rt: 1000008
axum 45, ct: 2000000, conn: 1000000, max_conn: 1000000, inner_mem: 2235.042M, outer_mem: 2532.879M, main_rt: 1, global_rt: 1000008
axum 46, ct: 2000000, conn: 1000000, max_conn: 1000000, inner_mem: 2235.042M, outer_mem: 2532.879M, main_rt: 1, global_rt: 1000008
axum 47, ct: 2000000, conn: 1000000, max_conn: 1000000, inner_mem: 2235.042M, outer_mem: 2532.879M, main_rt: 1, global_rt: 1000008
axum 48, ct: 2000000, conn: 1000000, max_conn: 1000000, inner_mem: 2235.042M, outer_mem: 2532.879M, main_rt: 1, global_rt: 1000008
axum 49, ct: 2000000, conn: 1000000, max_conn: 1000000, inner_mem: 2235.042M, outer_mem: 2532.879M, main_rt: 1, global_rt: 1000008
axum 50, ct: 2000000, conn: 1000000, max_conn: 1000000, inner_mem: 2235.042M, outer_mem: 2532.879M, main_rt: 1, global_rt: 1000008
axum 51, ct: 2000000, conn: 1000000, max_conn: 1000000, inner_mem: 2235.042M, outer_mem: 2532.879M, main_rt: 1, global_rt: 1000008
axum 52, ct: 2000000, conn: 1000000, max_conn: 1000000, inner_mem: 2235.042M, outer_mem: 2532.879M, main_rt: 1, global_rt: 1000008
axum 53, ct: 2000000, conn: 1000000, max_conn: 1000000, inner_mem: 2235.042M, outer_mem: 2532.879M, main_rt: 1, global_rt: 1000008
axum 54, ct: 2000000, conn: 873388, max_conn: 1000000, inner_mem: 2062.612M, outer_mem: 2553.004M, main_rt: 1, global_rt: 871597
axum 55, ct: 2000000, conn: 702935, max_conn: 1000000, inner_mem: 1772.213M, outer_mem: 2580.754M, main_rt: 1, global_rt: 701419
axum 56, ct: 2000000, conn: 481642, max_conn: 1000000, inner_mem: 1506.089M, outer_mem: 2630.254M, main_rt: 1, global_rt: 480889
axum 57, ct: 2000000, conn: 285573, max_conn: 1000000, inner_mem: 1309.204M, outer_mem: 2714.598M, main_rt: 1, global_rt: 282309
axum 58, ct: 2000000, conn: 131773, max_conn: 1000000, inner_mem: 1048.973M, outer_mem: 2729.223M, main_rt: 1, global_rt: 129783
axum 59, ct: 2000000, conn: 3418, max_conn: 1000000, inner_mem: 708.770M, outer_mem: 2736.723M, main_rt: 1, global_rt: 824
axum 60, ct: 2000000, conn: 0, max_conn: 1000000, inner_mem: 627.293M, outer_mem: 2736.723M, main_rt: 1, global_rt: 8
axum 61, ct: 2000000, conn: 0, max_conn: 1000000, inner_mem: 627.293M, outer_mem: 2736.723M, main_rt: 1, global_rt: 8
axum 62, ct: 2000000, conn: 0, max_conn: 1000000, inner_mem: 627.293M, outer_mem: 2736.723M, main_rt: 1, global_rt: 8
axum 63, ct: 2000000, conn: 0, max_conn: 1000000, inner_mem: 0.258M, outer_mem: 2339.508M, main_rt: 1, global_rt: 0
axum 64, ct: 2000000, conn: 0, max_conn: 1000000, inner_mem: 0.258M, outer_mem: 2339.508M, main_rt: 1, global_rt: 0
axum 65, ct: 2000000, conn: 0, max_conn: 1000000, inner_mem: 0.258M, outer_mem: 2339.508M, main_rt: 1, global_rt: 0
axum 66, ct: 2000000, conn: 0, max_conn: 1000000, inner_mem: 0.258M, outer_mem: 2339.508M, main_rt: 1, global_rt: 0
axum 67, ct: 2000000, conn: 0, max_conn: 1000000, inner_mem: 0.258M, outer_mem: 2339.508M, main_rt: 1, global_rt: 0
axum 68, ct: 2000000, conn: 0, max_conn: 1000000, inner_mem: 0.258M, outer_mem: 2339.508M, main_rt: 1, global_rt: 0
axum 69, ct: 2000000, conn: 0, max_conn: 1000000, inner_mem: 0.258M, outer_mem: 2339.508M, main_rt: 1, global_rt: 0
axum 70, ct: 2000000, conn: 0, max_conn: 1000000, inner_mem: 0.258M, outer_mem: 2339.508M, main_rt: 1, global_rt: 0
2、mimalloc-without-refresh
axum-demo pid is 2259
listening on 0.0.0.0:3000
axum 0, ct: 0, conn: 0, max_conn: 0, inner_mem: 0.258M, outer_mem: 7.875M, main_rt: 1, global_rt: 0
axum 1, ct: 0, conn: 0, max_conn: 0, inner_mem: 0.258M, outer_mem: 8.480M, main_rt: 1, global_rt: 0
axum 2, ct: 0, conn: 0, max_conn: 0, inner_mem: 0.258M, outer_mem: 9.355M, main_rt: 1, global_rt: 0
axum 3, ct: 0, conn: 0, max_conn: 0, inner_mem: 0.258M, outer_mem: 10.105M, main_rt: 1, global_rt: 0
axum 4, ct: 2358, conn: 2358, max_conn: 2358, inner_mem: 5.898M, outer_mem: 19.230M, main_rt: 1, global_rt: 2366
axum 5, ct: 195623, conn: 195623, max_conn: 195623, inner_mem: 465.840M, outer_mem: 484.844M, main_rt: 1, global_rt: 195631
axum 6, ct: 320341, conn: 320341, max_conn: 320341, inner_mem: 762.931M, outer_mem: 788.125M, main_rt: 1, global_rt: 320349
axum 7, ct: 482341, conn: 482341, max_conn: 482341, inner_mem: 1152.570M, outer_mem: 1188.777M, main_rt: 1, global_rt: 504142
axum 8, ct: 757306, conn: 757306, max_conn: 757306, inner_mem: 1901.621M, outer_mem: 1954.539M, main_rt: 1, global_rt: 782888
axum 9, ct: 1000000, conn: 1000000, max_conn: 1000000, inner_mem: 2242.541M, outer_mem: 2340.543M, main_rt: 1, global_rt: 1000008
axum 10, ct: 1000000, conn: 1000000, max_conn: 1000000, inner_mem: 2242.541M, outer_mem: 2340.543M, main_rt: 1, global_rt: 1000008
axum 11, ct: 1000000, conn: 1000000, max_conn: 1000000, inner_mem: 2242.541M, outer_mem: 2340.668M, main_rt: 1, global_rt: 1000008
axum 12, ct: 1000000, conn: 1000000, max_conn: 1000000, inner_mem: 2242.541M, outer_mem: 2340.793M, main_rt: 1, global_rt: 1000008
axum 13, ct: 1000000, conn: 1000000, max_conn: 1000000, inner_mem: 2242.541M, outer_mem: 2340.793M, main_rt: 1, global_rt: 1000008
axum 14, ct: 1000000, conn: 1000000, max_conn: 1000000, inner_mem: 2242.541M, outer_mem: 2340.793M, main_rt: 1, global_rt: 1000008
axum 15, ct: 1000000, conn: 1000000, max_conn: 1000000, inner_mem: 2242.541M, outer_mem: 2340.793M, main_rt: 1, global_rt: 1000008
axum 16, ct: 1000000, conn: 1000000, max_conn: 1000000, inner_mem: 2242.541M, outer_mem: 2340.793M, main_rt: 1, global_rt: 1000008
axum 17, ct: 1000000, conn: 1000000, max_conn: 1000000, inner_mem: 2242.541M, outer_mem: 2340.793M, main_rt: 1, global_rt: 1000008
axum 18, ct: 1000000, conn: 1000000, max_conn: 1000000, inner_mem: 2242.541M, outer_mem: 2340.793M, main_rt: 1, global_rt: 1000008
axum 19, ct: 1000000, conn: 997642, max_conn: 1000000, inner_mem: 2238.318M, outer_mem: 2336.062M, main_rt: 1, global_rt: 997650
axum 20, ct: 1000000, conn: 835549, max_conn: 1000000, inner_mem: 2064.168M, outer_mem: 2231.395M, main_rt: 1, global_rt: 833324
axum 21, ct: 1000000, conn: 681860, max_conn: 1000000, inner_mem: 1759.974M, outer_mem: 1824.879M, main_rt: 1, global_rt: 680530
axum 22, ct: 1000000, conn: 536398, max_conn: 1000000, inner_mem: 1512.444M, outer_mem: 1585.848M, main_rt: 1, global_rt: 536015
axum 23, ct: 1000000, conn: 298206, max_conn: 1000000, inner_mem: 1164.689M, outer_mem: 1244.891M, main_rt: 1, global_rt: 285720
axum 24, ct: 1000000, conn: 79740, max_conn: 1000000, inner_mem: 912.242M, outer_mem: 966.766M, main_rt: 1, global_rt: 75767
axum 25, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 672.618M, outer_mem: 778.508M, main_rt: 1, global_rt: 8
axum 26, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 664.043M, outer_mem: 778.359M, main_rt: 1, global_rt: 8
axum 27, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 664.043M, outer_mem: 778.359M, main_rt: 1, global_rt: 8
axum 28, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 664.043M, outer_mem: 778.359M, main_rt: 1, global_rt: 8
axum 29, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 0.258M, outer_mem: 731.422M, main_rt: 1, global_rt: 0
axum 30, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 0.258M, outer_mem: 731.422M, main_rt: 1, global_rt: 0
axum 31, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 0.258M, outer_mem: 731.422M, main_rt: 1, global_rt: 0
axum 32, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 0.258M, outer_mem: 731.422M, main_rt: 1, global_rt: 0
axum 33, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 0.258M, outer_mem: 731.422M, main_rt: 1, global_rt: 0
axum 34, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 0.258M, outer_mem: 731.266M, main_rt: 1, global_rt: 0
axum 35, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 0.258M, outer_mem: 731.391M, main_rt: 1, global_rt: 0
axum 36, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 0.258M, outer_mem: 731.391M, main_rt: 1, global_rt: 0
axum 37, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 0.258M, outer_mem: 731.391M, main_rt: 1, global_rt: 0
axum 38, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 0.258M, outer_mem: 731.391M, main_rt: 1, global_rt: 0
axum 39, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 0.318M, outer_mem: 617.980M, main_rt: 1, global_rt: 2
axum 40, ct: 1174121, conn: 174121, max_conn: 1000000, inner_mem: 441.919M, outer_mem: 842.766M, main_rt: 1, global_rt: 178585
axum 41, ct: 1376203, conn: 376203, max_conn: 1000000, inner_mem: 943.529M, outer_mem: 945.648M, main_rt: 1, global_rt: 381107
axum 42, ct: 1536470, conn: 536470, max_conn: 1000000, inner_mem: 1250.351M, outer_mem: 1283.625M, main_rt: 1, global_rt: 544540
axum 43, ct: 1808908, conn: 808908, max_conn: 1000000, inner_mem: 1902.623M, outer_mem: 1964.914M, main_rt: 1, global_rt: 813646
axum 44, ct: 2000000, conn: 1000000, max_conn: 1000000, inner_mem: 2238.791M, outer_mem: 2336.215M, main_rt: 1, global_rt: 1000008
axum 45, ct: 2000000, conn: 1000000, max_conn: 1000000, inner_mem: 2238.791M, outer_mem: 2335.840M, main_rt: 1, global_rt: 1000008
axum 46, ct: 2000000, conn: 1000000, max_conn: 1000000, inner_mem: 2238.791M, outer_mem: 2335.840M, main_rt: 1, global_rt: 1000008
axum 47, ct: 2000000, conn: 1000000, max_conn: 1000000, inner_mem: 2238.791M, outer_mem: 2335.660M, main_rt: 1, global_rt: 1000008
axum 48, ct: 2000000, conn: 1000000, max_conn: 1000000, inner_mem: 2238.791M, outer_mem: 2335.660M, main_rt: 1, global_rt: 1000008
axum 49, ct: 2000000, conn: 1000000, max_conn: 1000000, inner_mem: 2238.791M, outer_mem: 2335.660M, main_rt: 1, global_rt: 1000008
axum 50, ct: 2000000, conn: 1000000, max_conn: 1000000, inner_mem: 2238.791M, outer_mem: 2335.660M, main_rt: 1, global_rt: 1000008
axum 51, ct: 2000000, conn: 1000000, max_conn: 1000000, inner_mem: 2238.791M, outer_mem: 2335.660M, main_rt: 1, global_rt: 1000008
axum 52, ct: 2000000, conn: 1000000, max_conn: 1000000, inner_mem: 2238.791M, outer_mem: 2335.660M, main_rt: 1, global_rt: 1000008
axum 53, ct: 2000000, conn: 1000000, max_conn: 1000000, inner_mem: 2238.791M, outer_mem: 2335.660M, main_rt: 1, global_rt: 1000008
axum 54, ct: 2000000, conn: 1000000, max_conn: 1000000, inner_mem: 2238.791M, outer_mem: 2335.660M, main_rt: 1, global_rt: 1000008
axum 55, ct: 2000000, conn: 836494, max_conn: 1000000, inner_mem: 1999.376M, outer_mem: 2119.828M, main_rt: 1, global_rt: 832779
axum 56, ct: 2000000, conn: 651632, max_conn: 1000000, inner_mem: 1695.710M, outer_mem: 1820.117M, main_rt: 1, global_rt: 648517
axum 57, ct: 2000000, conn: 485491, max_conn: 1000000, inner_mem: 1389.168M, outer_mem: 1500.973M, main_rt: 1, global_rt: 484290
axum 58, ct: 2000000, conn: 245103, max_conn: 1000000, inner_mem: 1016.947M, outer_mem: 1136.078M, main_rt: 1, global_rt: 234059
axum 59, ct: 2000000, conn: 46343, max_conn: 1000000, inner_mem: 776.429M, outer_mem: 893.582M, main_rt: 1, global_rt: 44420
axum 60, ct: 2000000, conn: 0, max_conn: 1000000, inner_mem: 601.793M, outer_mem: 668.934M, main_rt: 1, global_rt: 8
axum 61, ct: 2000000, conn: 0, max_conn: 1000000, inner_mem: 601.793M, outer_mem: 668.809M, main_rt: 1, global_rt: 8
axum 62, ct: 2000000, conn: 0, max_conn: 1000000, inner_mem: 601.793M, outer_mem: 668.684M, main_rt: 1, global_rt: 8
axum 63, ct: 2000000, conn: 0, max_conn: 1000000, inner_mem: 601.793M, outer_mem: 668.684M, main_rt: 1, global_rt: 8
axum 64, ct: 2000000, conn: 0, max_conn: 1000000, inner_mem: 0.258M, outer_mem: 656.016M, main_rt: 1, global_rt: 0
axum 65, ct: 2000000, conn: 0, max_conn: 1000000, inner_mem: 0.258M, outer_mem: 656.016M, main_rt: 1, global_rt: 0
axum 66, ct: 2000000, conn: 0, max_conn: 1000000, inner_mem: 0.258M, outer_mem: 655.871M, main_rt: 1, global_rt: 0
axum 67, ct: 2000000, conn: 0, max_conn: 1000000, inner_mem: 0.258M, outer_mem: 655.871M, main_rt: 1, global_rt: 0
axum 68, ct: 2000000, conn: 0, max_conn: 1000000, inner_mem: 0.258M, outer_mem: 655.871M, main_rt: 1, global_rt: 0
axum 69, ct: 2000000, conn: 0, max_conn: 1000000, inner_mem: 0.258M, outer_mem: 655.871M, main_rt: 1, global_rt: 0
axum 70, ct: 2000000, conn: 0, max_conn: 1000000, inner_mem: 0.258M, outer_mem: 655.871M, main_rt: 1, global_rt: 0
3、default-with-refresh
axum-demo pid is 2849
listening on 0.0.0.0:3000
axum 0, ct: 0, conn: 0, max_conn: 0, inner_mem: 0.258M, outer_mem: 6.250M, main_rt: 1, global_rt: 0
axum 1, ct: 0, conn: 0, max_conn: 0, inner_mem: 0.258M, outer_mem: 6.559M, main_rt: 1, global_rt: 0
axum 2, ct: 0, conn: 0, max_conn: 0, inner_mem: 0.258M, outer_mem: 6.777M, main_rt: 1, global_rt: 0
axum 3, ct: 0, conn: 0, max_conn: 0, inner_mem: 0.258M, outer_mem: 6.805M, main_rt: 1, global_rt: 0
axum 4, ct: 0, conn: 0, max_conn: 0, inner_mem: 0.262M, outer_mem: 6.824M, main_rt: 1, global_rt: 2
axum 5, ct: 67033, conn: 67033, max_conn: 67033, inner_mem: 185.561M, outer_mem: 189.320M, main_rt: 1, global_rt: 68360
axum 6, ct: 211195, conn: 211195, max_conn: 211195, inner_mem: 514.785M, outer_mem: 540.012M, main_rt: 1, global_rt: 214611
axum 7, ct: 362115, conn: 362115, max_conn: 362115, inner_mem: 865.917M, outer_mem: 899.109M, main_rt: 1, global_rt: 369238
axum 8, ct: 571405, conn: 571405, max_conn: 571405, inner_mem: 1492.590M, outer_mem: 1567.328M, main_rt: 1, global_rt: 641803
axum 9, ct: 823134, conn: 823134, max_conn: 823134, inner_mem: 2045.258M, outer_mem: 2186.203M, main_rt: 1, global_rt: 884145
axum 10, ct: 1000000, conn: 1000000, max_conn: 1000000, inner_mem: 2242.541M, outer_mem: 2444.703M, main_rt: 1, global_rt: 1000008
axum 11, ct: 1000000, conn: 1000000, max_conn: 1000000, inner_mem: 2242.541M, outer_mem: 2444.703M, main_rt: 1, global_rt: 1000008
axum 12, ct: 1000000, conn: 1000000, max_conn: 1000000, inner_mem: 2242.541M, outer_mem: 2444.703M, main_rt: 1, global_rt: 1000008
axum 13, ct: 1000000, conn: 1000000, max_conn: 1000000, inner_mem: 2242.541M, outer_mem: 2444.828M, main_rt: 1, global_rt: 1000008
axum 14, ct: 1000000, conn: 1000000, max_conn: 1000000, inner_mem: 2242.541M, outer_mem: 2444.828M, main_rt: 1, global_rt: 1000008
axum 15, ct: 1000000, conn: 1000000, max_conn: 1000000, inner_mem: 2242.541M, outer_mem: 2444.828M, main_rt: 1, global_rt: 1000008
axum 16, ct: 1000000, conn: 1000000, max_conn: 1000000, inner_mem: 2242.541M, outer_mem: 2444.828M, main_rt: 1, global_rt: 1000008
axum 17, ct: 1000000, conn: 1000000, max_conn: 1000000, inner_mem: 2242.541M, outer_mem: 2444.828M, main_rt: 1, global_rt: 1000008
axum 18, ct: 1000000, conn: 1000000, max_conn: 1000000, inner_mem: 2242.541M, outer_mem: 2444.828M, main_rt: 1, global_rt: 1000008
axum 19, ct: 1000000, conn: 1000000, max_conn: 1000000, inner_mem: 2242.541M, outer_mem: 2444.828M, main_rt: 1, global_rt: 1000008
axum 20, ct: 1000000, conn: 940579, max_conn: 1000000, inner_mem: 2156.890M, outer_mem: 2456.828M, main_rt: 1, global_rt: 940594
axum 21, ct: 1000000, conn: 805314, max_conn: 1000000, inner_mem: 1923.350M, outer_mem: 2476.078M, main_rt: 1, global_rt: 803973
axum 22, ct: 1000000, conn: 654103, max_conn: 1000000, inner_mem: 1661.931M, outer_mem: 2480.578M, main_rt: 1, global_rt: 650122
axum 23, ct: 1000000, conn: 448379, max_conn: 1000000, inner_mem: 1349.720M, outer_mem: 2480.578M, main_rt: 1, global_rt: 446867
axum 24, ct: 1000000, conn: 264076, max_conn: 1000000, inner_mem: 1095.251M, outer_mem: 2493.828M, main_rt: 1, global_rt: 259210
axum 25, ct: 1000000, conn: 125734, max_conn: 1000000, inner_mem: 873.939M, outer_mem: 2532.328M, main_rt: 1, global_rt: 125749
axum 26, ct: 1000000, conn: 4260, max_conn: 1000000, inner_mem: 629.933M, outer_mem: 2554.203M, main_rt: 1, global_rt: 4275
axum 27, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 586.043M, outer_mem: 2455.984M, main_rt: 1, global_rt: 8
axum 28, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 586.043M, outer_mem: 2455.984M, main_rt: 1, global_rt: 8
axum 29, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 586.043M, outer_mem: 2455.984M, main_rt: 1, global_rt: 8
axum 30, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 0.347M, outer_mem: 1962.844M, main_rt: 1, global_rt: 0, refresh runtime on DefaultAlloc until 20M! cost 1.022712ms
axum 31, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 0.266M, outer_mem: 11.980M, main_rt: 1, global_rt: 0
axum 32, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 0.266M, outer_mem: 11.980M, main_rt: 1, global_rt: 0
axum 33, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 0.266M, outer_mem: 11.980M, main_rt: 1, global_rt: 0
axum 34, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 0.266M, outer_mem: 11.980M, main_rt: 1, global_rt: 0
axum 35, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 0.266M, outer_mem: 11.980M, main_rt: 1, global_rt: 0
axum 36, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 0.266M, outer_mem: 11.980M, main_rt: 1, global_rt: 0
axum 37, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 0.266M, outer_mem: 11.980M, main_rt: 1, global_rt: 0
axum 38, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 0.266M, outer_mem: 11.980M, main_rt: 1, global_rt: 0
axum 39, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 0.266M, outer_mem: 11.980M, main_rt: 1, global_rt: 0
axum 40, ct: 1121303, conn: 121303, max_conn: 1000000, inner_mem: 319.137M, outer_mem: 336.855M, main_rt: 1, global_rt: 125834
axum 41, ct: 1309984, conn: 309984, max_conn: 1000000, inner_mem: 823.280M, outer_mem: 852.605M, main_rt: 1, global_rt: 342849
axum 42, ct: 1559032, conn: 559032, max_conn: 1000000, inner_mem: 1360.795M, outer_mem: 1436.855M, main_rt: 1, global_rt: 574630
axum 43, ct: 1791651, conn: 791651, max_conn: 1000000, inner_mem: 1900.792M, outer_mem: 2052.355M, main_rt: 1, global_rt: 841903
axum 44, ct: 1977056, conn: 977056, max_conn: 1000000, inner_mem: 2193.794M, outer_mem: 2372.980M, main_rt: 1, global_rt: 978721
axum 45, ct: 2000000, conn: 1000000, max_conn: 1000000, inner_mem: 2235.049M, outer_mem: 2417.105M, main_rt: 1, global_rt: 1000008
axum 46, ct: 2000000, conn: 1000000, max_conn: 1000000, inner_mem: 2235.049M, outer_mem: 2417.105M, main_rt: 1, global_rt: 1000008
axum 47, ct: 2000000, conn: 1000000, max_conn: 1000000, inner_mem: 2235.049M, outer_mem: 2417.105M, main_rt: 1, global_rt: 1000008
axum 48, ct: 2000000, conn: 1000000, max_conn: 1000000, inner_mem: 2235.049M, outer_mem: 2417.105M, main_rt: 1, global_rt: 1000008
axum 49, ct: 2000000, conn: 1000000, max_conn: 1000000, inner_mem: 2235.049M, outer_mem: 2417.105M, main_rt: 1, global_rt: 1000008
axum 50, ct: 2000000, conn: 1000000, max_conn: 1000000, inner_mem: 2235.049M, outer_mem: 2417.105M, main_rt: 1, global_rt: 1000008
axum 51, ct: 2000000, conn: 1000000, max_conn: 1000000, inner_mem: 2235.049M, outer_mem: 2417.105M, main_rt: 1, global_rt: 1000008
axum 52, ct: 2000000, conn: 1000000, max_conn: 1000000, inner_mem: 2235.049M, outer_mem: 2417.105M, main_rt: 1, global_rt: 1000008
axum 53, ct: 2000000, conn: 1000000, max_conn: 1000000, inner_mem: 2235.049M, outer_mem: 2417.105M, main_rt: 1, global_rt: 1000008
axum 54, ct: 2000000, conn: 1000000, max_conn: 1000000, inner_mem: 2235.049M, outer_mem: 2417.105M, main_rt: 1, global_rt: 1000008
axum 55, ct: 2000000, conn: 900346, max_conn: 1000000, inner_mem: 2096.700M, outer_mem: 2453.855M, main_rt: 1, global_rt: 897519
axum 56, ct: 2000000, conn: 711036, max_conn: 1000000, inner_mem: 1836.852M, outer_mem: 2502.855M, main_rt: 1, global_rt: 709887
axum 57, ct: 2000000, conn: 538257, max_conn: 1000000, inner_mem: 1587.075M, outer_mem: 2529.605M, main_rt: 1, global_rt: 535972
axum 58, ct: 2000000, conn: 343601, max_conn: 1000000, inner_mem: 1362.405M, outer_mem: 2566.566M, main_rt: 1, global_rt: 340818
axum 59, ct: 2000000, conn: 174574, max_conn: 1000000, inner_mem: 1187.495M, outer_mem: 2650.773M, main_rt: 1, global_rt: 170441
axum 60, ct: 2000000, conn: 16180, max_conn: 1000000, inner_mem: 1035.920M, outer_mem: 2719.148M, main_rt: 1, global_rt: 16195
axum 61, ct: 2000000, conn: 0, max_conn: 1000000, inner_mem: 776.194M, outer_mem: 2727.398M, main_rt: 1, global_rt: 8
axum 62, ct: 2000000, conn: 0, max_conn: 1000000, inner_mem: 754.051M, outer_mem: 2727.398M, main_rt: 1, global_rt: 8
axum 63, ct: 2000000, conn: 0, max_conn: 1000000, inner_mem: 754.051M, outer_mem: 2727.398M, main_rt: 1, global_rt: 8
axum 64, ct: 2000000, conn: 0, max_conn: 1000000, inner_mem: 754.051M, outer_mem: 2727.398M, main_rt: 1, global_rt: 8
axum 65, ct: 2000000, conn: 0, max_conn: 1000000, inner_mem: 754.051M, outer_mem: 2727.398M, main_rt: 1, global_rt: 8
axum 66, ct: 2000000, conn: 0, max_conn: 1000000, inner_mem: 0.345M, outer_mem: 2151.961M, main_rt: 1, global_rt: 0, refresh runtime on DefaultAlloc until 20M! cost 721.269µs
axum 67, ct: 2000000, conn: 0, max_conn: 1000000, inner_mem: 0.344M, outer_mem: 62.145M, main_rt: 1, global_rt: 0, refresh runtime on DefaultAlloc until 20M! cost 823.072µs
axum 68, ct: 2000000, conn: 0, max_conn: 1000000, inner_mem: 0.266M, outer_mem: 62.145M, main_rt: 1, global_rt: 0, refresh runtime on DefaultAlloc until 20M! cost 870.368µs
axum 69, ct: 2000000, conn: 0, max_conn: 1000000, inner_mem: 0.344M, outer_mem: 62.145M, main_rt: 1, global_rt: 0, refresh runtime on DefaultAlloc until 20M! cost 915.286µs
axum 70, ct: 2000000, conn: 0, max_conn: 1000000, inner_mem: 0.315M, outer_mem: 62.145M, main_rt: 1, global_rt: 0, refresh runtime on DefaultAlloc until 20M! cost 866.946µs
4、mimalloc-with-refresh
axum-demo pid is 3021
listening on 0.0.0.0:3000
axum 0, ct: 0, conn: 0, max_conn: 0, inner_mem: 0.258M, outer_mem: 7.500M, main_rt: 1, global_rt: 0
axum 1, ct: 0, conn: 0, max_conn: 0, inner_mem: 0.258M, outer_mem: 8.484M, main_rt: 1, global_rt: 0
axum 2, ct: 0, conn: 0, max_conn: 0, inner_mem: 0.258M, outer_mem: 9.609M, main_rt: 1, global_rt: 0
axum 3, ct: 0, conn: 0, max_conn: 0, inner_mem: 0.258M, outer_mem: 10.359M, main_rt: 1, global_rt: 0
axum 4, ct: 0, conn: 0, max_conn: 0, inner_mem: 0.258M, outer_mem: 10.484M, main_rt: 1, global_rt: 0
axum 5, ct: 35355, conn: 35355, max_conn: 35355, inner_mem: 88.450M, outer_mem: 105.109M, main_rt: 1, global_rt: 37147
axum 6, ct: 138968, conn: 138968, max_conn: 138968, inner_mem: 329.454M, outer_mem: 350.250M, main_rt: 1, global_rt: 142594
axum 7, ct: 384539, conn: 384539, max_conn: 384539, inner_mem: 962.115M, outer_mem: 983.426M, main_rt: 1, global_rt: 393629
axum 8, ct: 606163, conn: 606163, max_conn: 606163, inner_mem: 1408.331M, outer_mem: 1450.270M, main_rt: 1, global_rt: 617051
axum 9, ct: 876509, conn: 876509, max_conn: 876509, inner_mem: 2030.256M, outer_mem: 2090.184M, main_rt: 1, global_rt: 881387
axum 10, ct: 1000000, conn: 1000000, max_conn: 1000000, inner_mem: 2235.041M, outer_mem: 2333.586M, main_rt: 1, global_rt: 1000008
axum 11, ct: 1000000, conn: 1000000, max_conn: 1000000, inner_mem: 2235.041M, outer_mem: 2333.836M, main_rt: 1, global_rt: 1000008
axum 12, ct: 1000000, conn: 1000000, max_conn: 1000000, inner_mem: 2235.041M, outer_mem: 2333.836M, main_rt: 1, global_rt: 1000008
axum 13, ct: 1000000, conn: 1000000, max_conn: 1000000, inner_mem: 2235.041M, outer_mem: 2333.836M, main_rt: 1, global_rt: 1000008
axum 14, ct: 1000000, conn: 1000000, max_conn: 1000000, inner_mem: 2235.041M, outer_mem: 2333.652M, main_rt: 1, global_rt: 1000008
axum 15, ct: 1000000, conn: 1000000, max_conn: 1000000, inner_mem: 2235.041M, outer_mem: 2333.652M, main_rt: 1, global_rt: 1000008
axum 16, ct: 1000000, conn: 1000000, max_conn: 1000000, inner_mem: 2235.041M, outer_mem: 2333.652M, main_rt: 1, global_rt: 1000008
axum 17, ct: 1000000, conn: 1000000, max_conn: 1000000, inner_mem: 2235.041M, outer_mem: 2333.652M, main_rt: 1, global_rt: 1000008
axum 18, ct: 1000000, conn: 1000000, max_conn: 1000000, inner_mem: 2235.041M, outer_mem: 2333.652M, main_rt: 1, global_rt: 1000008
axum 19, ct: 1000000, conn: 1000000, max_conn: 1000000, inner_mem: 2235.041M, outer_mem: 2333.777M, main_rt: 1, global_rt: 1000008
axum 20, ct: 1000000, conn: 985848, max_conn: 1000000, inner_mem: 2217.086M, outer_mem: 2328.094M, main_rt: 1, global_rt: 977496
axum 21, ct: 1000000, conn: 862314, max_conn: 1000000, inner_mem: 2014.693M, outer_mem: 2106.859M, main_rt: 1, global_rt: 860119
axum 22, ct: 1000000, conn: 655363, max_conn: 1000000, inner_mem: 1707.808M, outer_mem: 1875.473M, main_rt: 1, global_rt: 655378
axum 23, ct: 1000000, conn: 459518, max_conn: 1000000, inner_mem: 1400.007M, outer_mem: 1536.480M, main_rt: 1, global_rt: 459533
axum 24, ct: 1000000, conn: 215690, max_conn: 1000000, inner_mem: 1082.380M, outer_mem: 1249.082M, main_rt: 1, global_rt: 214247
axum 25, ct: 1000000, conn: 33991, max_conn: 1000000, inner_mem: 846.364M, outer_mem: 948.512M, main_rt: 1, global_rt: 24484
axum 26, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 705.293M, outer_mem: 712.348M, main_rt: 1, global_rt: 8
axum 27, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 705.293M, outer_mem: 712.086M, main_rt: 1, global_rt: 8
axum 28, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 705.293M, outer_mem: 711.949M, main_rt: 1, global_rt: 8
axum 29, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 705.293M, outer_mem: 711.812M, main_rt: 1, global_rt: 8
axum 30, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 0.345M, outer_mem: 674.520M, main_rt: 1, global_rt: 0, refresh runtime on MiMalloc until 20M! cost 1.190306ms
axum 31, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 0.347M, outer_mem: 298.633M, main_rt: 1, global_rt: 0, refresh runtime on MiMalloc until 20M! cost 713.71µs
axum 32, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 0.266M, outer_mem: 13.512M, main_rt: 1, global_rt: 0
axum 33, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 0.266M, outer_mem: 14.387M, main_rt: 1, global_rt: 0
axum 34, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 0.266M, outer_mem: 14.387M, main_rt: 1, global_rt: 0
axum 35, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 0.266M, outer_mem: 14.262M, main_rt: 1, global_rt: 0
axum 36, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 0.266M, outer_mem: 14.387M, main_rt: 1, global_rt: 0
axum 37, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 0.266M, outer_mem: 14.387M, main_rt: 1, global_rt: 0
axum 38, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 0.266M, outer_mem: 14.387M, main_rt: 1, global_rt: 0
axum 39, ct: 1000000, conn: 0, max_conn: 1000000, inner_mem: 0.266M, outer_mem: 14.387M, main_rt: 1, global_rt: 0
axum 40, ct: 1022144, conn: 22144, max_conn: 1000000, inner_mem: 58.355M, outer_mem: 75.668M, main_rt: 1, global_rt: 24549
axum 41, ct: 1243949, conn: 243949, max_conn: 1000000, inner_mem: 572.814M, outer_mem: 586.391M, main_rt: 1, global_rt: 245193
axum 42, ct: 1459547, conn: 459547, max_conn: 1000000, inner_mem: 1170.676M, outer_mem: 1156.102M, main_rt: 1, global_rt: 492212
axum 43, ct: 1688070, conn: 688070, max_conn: 1000000, inner_mem: 1612.515M, outer_mem: 1604.105M, main_rt: 1, global_rt: 688105
axum 44, ct: 1910744, conn: 910744, max_conn: 1000000, inner_mem: 2099.287M, outer_mem: 2171.328M, main_rt: 1, global_rt: 940210
axum 45, ct: 2000000, conn: 1000000, max_conn: 1000000, inner_mem: 2210.674M, outer_mem: 2312.785M, main_rt: 1, global_rt: 1000008
axum 46, ct: 2000000, conn: 1000000, max_conn: 1000000, inner_mem: 2210.674M, outer_mem: 2312.910M, main_rt: 1, global_rt: 1000008
axum 47, ct: 2000000, conn: 1000000, max_conn: 1000000, inner_mem: 2210.674M, outer_mem: 2313.035M, main_rt: 1, global_rt: 1000008
axum 48, ct: 2000000, conn: 1000000, max_conn: 1000000, inner_mem: 2210.674M, outer_mem: 2313.035M, main_rt: 1, global_rt: 1000008
axum 49, ct: 2000000, conn: 1000000, max_conn: 1000000, inner_mem: 2210.674M, outer_mem: 2313.035M, main_rt: 1, global_rt: 1000008
axum 50, ct: 2000000, conn: 1000000, max_conn: 1000000, inner_mem: 2210.674M, outer_mem: 2313.035M, main_rt: 1, global_rt: 1000008
axum 51, ct: 2000000, conn: 1000000, max_conn: 1000000, inner_mem: 2210.674M, outer_mem: 2313.035M, main_rt: 1, global_rt: 1000008
axum 52, ct: 2000000, conn: 1000000, max_conn: 1000000, inner_mem: 2210.674M, outer_mem: 2313.035M, main_rt: 1, global_rt: 1000008
axum 53, ct: 2000000, conn: 1000000, max_conn: 1000000, inner_mem: 2210.674M, outer_mem: 2312.906M, main_rt: 1, global_rt: 1000008
axum 54, ct: 2000000, conn: 1000000, max_conn: 1000000, inner_mem: 2210.674M, outer_mem: 2312.906M, main_rt: 1, global_rt: 1000008
axum 55, ct: 2000000, conn: 986024, max_conn: 1000000, inner_mem: 2193.908M, outer_mem: 2309.086M, main_rt: 1, global_rt: 985245
axum 56, ct: 2000000, conn: 767038, max_conn: 1000000, inner_mem: 1835.938M, outer_mem: 1934.230M, main_rt: 1, global_rt: 764543
axum 57, ct: 2000000, conn: 563257, max_conn: 1000000, inner_mem: 1486.607M, outer_mem: 1598.703M, main_rt: 1, global_rt: 558215
axum 58, ct: 2000000, conn: 339905, max_conn: 1000000, inner_mem: 1156.538M, outer_mem: 1261.723M, main_rt: 1, global_rt: 338950
axum 59, ct: 2000000, conn: 132751, max_conn: 1000000, inner_mem: 773.141M, outer_mem: 852.016M, main_rt: 1, global_rt: 129134
axum 60, ct: 2000000, conn: 0, max_conn: 1000000, inner_mem: 524.926M, outer_mem: 576.590M, main_rt: 1, global_rt: 8
axum 61, ct: 2000000, conn: 0, max_conn: 1000000, inner_mem: 524.926M, outer_mem: 576.465M, main_rt: 1, global_rt: 8
axum 62, ct: 2000000, conn: 0, max_conn: 1000000, inner_mem: 524.926M, outer_mem: 576.324M, main_rt: 1, global_rt: 8
axum 63, ct: 2000000, conn: 0, max_conn: 1000000, inner_mem: 524.926M, outer_mem: 576.324M, main_rt: 1, global_rt: 8
axum 64, ct: 2000000, conn: 0, max_conn: 1000000, inner_mem: 524.926M, outer_mem: 576.324M, main_rt: 1, global_rt: 8
axum 65, ct: 2000000, conn: 0, max_conn: 1000000, inner_mem: 0.345M, outer_mem: 549.789M, main_rt: 1, global_rt: 0, refresh runtime on MiMalloc until 20M! cost 543.071µs
axum 66, ct: 2000000, conn: 0, max_conn: 1000000, inner_mem: 0.347M, outer_mem: 432.996M, main_rt: 1, global_rt: 0, refresh runtime on MiMalloc until 20M! cost 980.94µs
axum 67, ct: 2000000, conn: 0, max_conn: 1000000, inner_mem: 0.347M, outer_mem: 27.895M, main_rt: 1, global_rt: 0, refresh runtime on MiMalloc until 20M! cost 930.906µs
axum 68, ct: 2000000, conn: 0, max_conn: 1000000, inner_mem: 0.266M, outer_mem: 13.184M, main_rt: 1, global_rt: 0
axum 69, ct: 2000000, conn: 0, max_conn: 1000000, inner_mem: 0.266M, outer_mem: 13.809M, main_rt: 1, global_rt: 0
axum 70, ct: 2000000, conn: 0, max_conn: 1000000, inner_mem: 0.266M, outer_mem: 13.934M, main_rt: 1, global_rt: 0
评论区
写评论使用 musl 编译好像没这问题
分配器不立即归还系统我理解是常态,无论 pt/je/tc malloc 都一样
issue现场大佬们提了可以用
unsafe { libc::malloc_trim(0); }
来刷新内存。测试
unsafe { libc::malloc_trim(0); }
的日志回复在issue里,论坛就不贴了。理论上可以通过
std::alloc
里相关的方法来追踪,但我看hyper/axum/warp的社区里内存泄漏的issue挂了好几年了,贡献者早就追踪过了,目前也没人有啥结论。我这个文章里刷新运行时也只能算是种比较trick的方法,如果真放到实践里也不知道会有什么别的问题。--
👇
zqlpaopao: 这个问题很多地方在讨论 怎么能监控或者定位内存到底在哪里呢
是的,而且这种提法已经有点委婉了,目前来看如果不上点手段的话基本是不可能回收的,出了问题放运维那里基本就可以定义成内存泄漏了。
--
👇
bestgopher: 牛!是不是释放了操作系统没立即回收?
这个问题很多地方在讨论 怎么能监控或者定位内存到底在哪里呢
牛!是不是释放了操作系统没立即回收?