As the cpu NUMA nodes and cpu sockets increasing. The Ceph daemons deployed in different sockets have different performance. There are four NUMA nodes in ARM server(based on KunPeng 920) with 2 cpu sockets.Each socket has direct pcie slots connecting Network Devices and Storage Devices. Under the circumstance, there is Non-Uniform IO Access between cpu and IO device. We needs a multi-NUMA affinity solution for Ceph.In Ceph, there are MDS,RGW,OSD which needing to be binded to NUMA node, especially for OSDs. We use 3 servers, every server have 2 cpu socket, 1 NVME SSD and 2 NICs for each socket. On each server,we divide OSDs into 4 groups according to NUMA nodes. OSDs in the same socket share the SSD and NICs.We set SSD's and NIC's irq affinity to the local socket. To avoid the memory access cross sockets which is caused by network's RX/TX. We set different IP for different socket NIC and use policy route to isolate network traffic cross sockets. Finally we let the OSDs using all of local socket resource(memroy,ssd,nic). Our experiment show lower latency and throughput improvement.
