Memory Management

Allocation

Upon boot time, in arch_setup_free_memory(), OSv discovers how much physical memory is available by reading ent820 entries and then linearly maps the identified memory ranges by calling memory::free_initial_memory_range(). So for example, given 100MB from the host, OSv would find a single memory range starting at wherever loader.elf ends - roughly 12MB offset and ending at 100MB (roughly ~88MB big). Once the memory is set up, all memory ranges are ultimately registered in memory::free_page_ranges of type page_range_allocator that effectively tracks all used/free physical memory and implements lowest level memory allocation logic. At this level, memory is tracked/allocated/freed in 4K chunks (pages) aligned at 0x000 addresses. TODO: Describe key fields - _free_huge and _free.

[TODO - Show map ranges from gdb]

From this point on OSv is ready to handle "malloc/free" family and memory::alloc_page()/free_page() calls by drawing/releasing memory from/to free_page_ranges in form of page_range objects (see methods page_range_allocator::alloc(), alloc_aligned() and free()) and mapping to virtual address ranges. However until much later when SMP is enabled (multiple vCPUs are fully activated), the allocations would be handled at a different granularity than after SMP is on. In addition in the first phase (pre-SMP enabled) the allocations draw pages directly from the free_page_ranges object, whereas after SMP is enabled they draw memory from L1/L2 pools. There are as many L1 pools as vCPUs (per-cpu construct) and a single global L2 pool. The L1 pools draw pages from the L2 pool which in turn draws page ranges from free_page_ranges. Both L1 and L2 pools operate at page size level and implement low/high watermark algorithm (for example L1 pools keep at least 128 pages of memory available).

It is also worth noting that most malloc functions (except malloc_large()) end up calling std_malloc() (see https://github.com/cloudius-systems/osv/blob/186779b2e477815bbcea8ccff6ba26a7e21cea09/core/mempool.cc#L1544-L1565) that allocates virtual memory in different ways depending on whether we are in pre/post-SMP enabled mode and depending on the size of the memory request. The sizes ranges are:

• x <= 1024 (page size/4)
• 1024 < x <= 4096
• x > 4096

If we are in SMP-enabled mode and requested size is less or equal 1024 bytes, the allocation is going to be delegated to malloc pools (see https://github.com/cloudius-systems/osv/blob/186779b2e477815bbcea8ccff6ba26a7e21cea09/core/mempool.cc#L177-L353). Malloc pools are setup per-CPU and dedicated to specific size range (2^(k-1) < x <=2^k where k is less or equal 10). The way std_malloc() handles <= 4K allocations directly impacts varying degrees of underlying physical memory utilization. For example, any request above 1024 bytes will use the whole page and in worst-case scenario waste 3K of physical memory. Similarly, malloc pool allocations in worst-case scenarios may waste up to half of 2^k-1 segment size.

The malloc_large/free_large() calls draw memory directly free_page_range in both pre- and post-SMP-enabled phases.

Mapping

• Page tables
• Code in mmu.cc