> For the complete documentation index, see [llms.txt](https://sliu583.gitbook.io/blog/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://sliu583.gitbook.io/blog/persistence/index/hardware/an-empirical-guide-to-the-behavior-and-use-of-scalable-persistent-memory.md).

# An Empirical Guide to the Behavior and Use of Scalable Persistent Memory

### Presentation 

* Optane DIMMs 
  * Not just slow dense DRAM 
  * Slower media 
    * More complex architecture 
      * Second-order performance anomalies 
        * Fundamentally different 
* Outline 
  * Background 
  * Basics: Optane DIMM performance 
  * Lessons: Optane DIMM best practices 
  * Conclusion 

#### Background: Optane in the machine 

* iMC: integrated memory controller 
* use optane 
  * *Memory mode* : use Optane to expand main memory capacity without persistence 
    * Combine Optane DIMM with a conventional DRAM DIMM on the same memory channel that serves as a direct-mapped cache for the NVDIMM 
  * *App Direct mode* : provides persistence and does not use a DRAM cache 

![Focus on the right ](/files/-MjHjEtOMVee3NYC2C2L)

* Optane controller: merging 4 cache lines into 256B block and issuing that block to media 
  * Buffer: merging 
* AIT: address interaction table 
  * Resides in optane media, but also cache on device dram 
* ADR: 
  * If power failure, then we have enough stand-up power to flush the write pending queue to media 
    * The ADR domain does not include the processor caches, so stores are only persistent once they reach WPQs 
  * WPQ: write pending queue 
  * The iMC communicates wih the Optane DIMM using the DDR-T interface in cache-line (64-byte) granularity
    * Has to do with the physical features of how to data is laid out 

![ADR ](/files/-MjHjlJ5IbaHbXwBvFGM)

![](/files/-MjHkCuevfTmw9SkKI09)

* First 4K: optane DIMM 1
* Next 4K: optane DIMM 4 
* Touch all six optane DIMM 

#### Basics: our approach 

![Microbenchmark ](/files/-MjHkbIMcxX_tcl7o-ME)

![](/files/-MjHl1xV8wtqaZu55U7c)

* Sequentially: no access amplification 
  * In contrast, random, 3x as small 
* Write 
  * Is acknowledged as soon as it hits the on socket integrated memory controller 
    * Not measure all the way to media 
      * Not possible to measure this? 
    * Writing to the cache + flushing it out 
      * Write to cache is the same for both cases 
      * Flush

![](/files/-MjHlSnpU9rYGJ3cW_yJ)

* DRAM
  * More cores, performance goes up 
* NI: non-interleave 
  * Read from single DIMM 
  * Bandwidth low 
  * Speculate: slow media, contentions, extra delay because of this contention 
* Interleave
  * Write: saturate bandwidth and contention is growing worse 
    * 3 cores and slow down 
* Use all optane dimms, interleave reads (scale well), efficiently
* Write (max out) constant with respect to thread counts 

![](/files/-MjHlriPLO2vizTQjtin)

* Using the block size efficiently 

![](/files/-MjHlyg8WSScyDi9bfH2)

* Read: Peak bandwidth at 512B, but valley at 4K (weird contention)
  * File system use this 4KB size 

#### Lessons: what are Optane best practices? 

* Avoid small random accesses 
* Use ntstores for large writes 
* Limit threads accessing one NVDIMM
* Avoid mixed and multi-threaded NUMA accesses

**Lesson 1: avoid small random accesses**  

![](/files/-MjHmSUE2BsNjcZRBWEs)

* Small accesses: smaller than 256B, lose bandwidth 

![](/files/-MjHmbbp7QIj2MORrjjP)

* Vary the working set size 
  * How large this on-device buffer is? About 16K 

![](/files/-MjHmtvKkoe5uPVbIDCG)

**Lesson 2: Use ntstores for large writes** 

![](/files/-MjHnAiDnyTClgNmAmcf)

* ntstore (non-temporal store): bypass the cache hierarchy and issue directly to the backing DIMM
* store + clwb: doing Aa store and then evicting that cache line or cleaning the cache line using a cache line write back (CLWB)
  * Con: Lost bandwidth 
  * Preserve the sequential access if all possible 
  * Doing a read plus a write, using double the bandwidth 
* store: trickle out from the cache into media  
  * Con: Lost locality 
  * Cache is going to evict the cache line when it decides to, not optimize for evicting things in a sequential pattern (i.e. introduce randomness) --> terrible device utilization 

![](/files/-MjHoIzJN9gmRnzSlylM)

![](/files/-MjHoQTtu_0TKFQP07Zp)

**Lesson #3: Limit threads accessing one NVDIMM** 

* Contention at Optane Buffer: merging adjacent accesses 
* Contention at iMC 

Contention at Optane Buffer 

![](/files/-MjHqSJbGyMBm2DEjd3G)

* Multiple threads: thrash on this on-device cache 
  * What would otherwise can be sequential access that can be merged is no longer the case 
  * All threads are putting their own writes, lose the opportunity to merge  
  * Read 
    * Spike: max bandwidth 
    * Add more: thrashing the cache 

![](/files/-MjHr-5XKmcXar9VOZ-X)

![](/files/-MjHr3DmWsvU_3E9LiQ8)

* Multiple threads - multiple DIMM - clogging the single DIMM (bandwidth falls down)
* Ends up with stall 

![](/files/-MjHwOOI_LTaSy4TrUtj)

* Exactly interleave size: burst of the accesses (hit the memory controller and the same DIMM) 

![](/files/-MjHwTXNw0aOgPthRt2J)

**Lesson #4: avoid mixed and multi-threaded NUMA accesses**

![](/files/-MjHwcL6ruUtXuPDnwVO)

![](/files/-MjHwkgzuP-I6RvsScBc)