Go 1.3垃圾收集器不会将服务器内存释放回系统 [英] Go 1.3 Garbage collector not releasing server memory back to system

问题描述

我们编写了最简单的TCP服务器（使用次要日志记录）来检查内存占用情况（请参阅下面的tcp-server.go）。

服务器只接受连接并什么也没做。它使用Go版本go1.3 linux / amd64在Ubuntu 12.04.4 LTS服务器（内核3.2.0-61-generic）上运行。

$ b

附加的基准测试程序（pulse.go）在这个例子中创建10k个连接，在30秒后断开它们，重复这个循环三次，然后不断重复1k连接/断开的小脉冲。用于测试的命令是./pulse -big = 10000 -bs = 30。

第一个附加图形是通过记录runtime.ReadMemStats获得的，当客户端数量改变了500的倍数，第二个图表是服务器进程的顶部看到的RES内存大小。

服务器以小于1.6KB的记忆。然后通过10k连接的大脉冲设置存储器，大小约60MB（如上图所示），或ReadMemStats所示的大约16MBSystemMemory。正如预期的那样，当10K脉冲结束时，正在使用的内存下降，最终程序开始将内存释放回操作系统，如灰色已释放内存行所示。

问题在于系统内存（相应地，由top看到的RES内存）从不显着下降（尽管它在第二个图中看到有点下降）。

我们预计在10K脉冲结束后，内存将继续释放，直到RES大小为处理每个1k脉冲所需的最小值（即8m RES由top和由runtime.ReadMemStats报告的2MB使用中所看到）。相反，RES保持在56MB左右，使用率永远不会从60MB的最高值下降。

我们希望确保不定期流量的可扩展性，偶尔会出现峰值以及能够在同一个盒子上运行多个服务器，这些服务器在不同的时间有尖峰。有没有办法有效地确保尽可能多的内存在合理的时间内被释放回系统？ .stack.imgur.com / 1g7Rk.pngalt =第一张图>

代码 https://gist.github.com/eugene-bulkin/e8d690b4db144f468bc5 ：

server.go：

package main

import（
net
log
runtime
sync
）
var m sync.Mutex
var num_clients = 0
var cycle = 0

func printMem（）{
var ms runtime.MemStats
runtime.ReadMemStats（& ms）
log.Printf（Cycle＃％3d：％5d clients | System：％8d Inuse：％8d Released：％8d Objects：％6d \\\
，cycle，num_clients，ms.HeapSys，ms.HeapInuse，ms.HeapReleased，ms.Hea pObjects）
}

func handleConnection（conn net.Conn）{
//log.Println(\"Accepted connection：，conn.RemoteAddr（））
m.Lock（）
num_clients ++
if num_clients％500 == 0 {
printMem（）
}
m.Unlock（）
buffer：= $ {$ b $，err：= conn.Read（buffer）
如果err！= nil {
//log.Println（丢失的连接：，conn.RemoteAddr（））
err：= conn.Close（）
if err！= nil {
log.Println（Connection close error：，err）
}
m.Lock（）
num_clients -
if num_clients％500 == 0 {
printMem（）
}
if num_clients == 0 {
cycle ++
}
m.Unlock（）
break
}
}
}

func main（）{
printMem（）
循环++
监听器，err：= net.Listen（tcp，：3033）
if err！= nil {
log.Fatal（Could not listen。）
}
for {
conn，err：= listener.Accept（）
if err！= nil {
log.Println（Could not listen to client：，err）
continue

go handleConnection（conn）
}
}


pulse.go：

package main

import（
flag
net
sync
log
time
）

var（
numBig = flag.Int（big，4000，大脉冲中的连接数量）
bigIters = flag.Int（i，3，大脉冲的迭代次数）
bigSep = flag.Int（bs，5，大脉冲之间的秒数）
numSmall = flag.Int（small，1000，小脉冲中的连接数量）
smallSep = flag.Int（ss，20，小脉冲之间的秒数）
linger = flag.Int（l，4，连接应断开多久才会断开连接）
）

var m sync.Mutex

var active_conns = 0
var c onnections = make（map [net.Conn] bool）

func pulse（n int，linger int）{
var wg sync.WaitGroup

log.Printf （连接％d个客户... \\\
，n）
为i：= 0;我< N; i ++ {
wg.Add（1）
go func（）{
m.Lock（）
推迟m.Unlock（）
延迟wg.Done（）
active_conns ++
conn，err：= net.Dial（tcp，：3033）
if err！= nil {
log.Panicln（无法连接： ，err）
return
}
connections [conn] = true
}（）
}
wg.Wait（）
if len（连接）！= n {
log.Fatalf（无法连接所有％d客户端。），n）
}
log.Printf（Connected ％d客户端。\ n，n）
time.Sleep（time.Duration（linger）* time.Second）
for conn：=范围连接{
active_conns- -
err：= conn.Close（）
如果err！= nil {
log.Panicln（无法关闭连接：，错误）
conn = nil


delete（connections，conn）
conn = nil
}
if len（connections）> 0 {
log.Fatalf（无法断开所有％d个客户端[％d仍然]。\ n，n，len（连接））
}
log。 printf（断开连接的％d个客户端。），n）
}

func main（）{
flag.Parse（）
for我：= 0;我< * bigIters; {
pulse（* numBig，* linger）
time.Sleep（time.Duration（* bigSep）* time.Second）
}
（* numSmall，* linger）
time.Sleep（time.Duration（* smallSep）* time.Second）
}
}
$ b

$ b

https://groups.google.com/forum/# ！topic / Golang-Nuts / vfmd6zaRQVs

堆被释放，您可以使用runtime.ReadMemStats（）来检查这个问题，
，但进程虚拟地址空间不缩小 - 即，您的
程序不会将内存返回给操作系统。在基于Unix的
平台上，我们使用系统调用来告诉操作系统它
可以回收堆中未使用的部分，该设备在Windows平台上不可用
。

但是你不在Windows上，对吗？

好的，这个线程不太确定，但它表示：

https： //groups.google.com/forum/#!topic/golang-nuts/MC2hWpuT7Xc

据我所知，内存在GC被标记为
后大约5分钟后返回操作系统。 GC每隔两分钟运行一次，如果不是由内存使用增加触发的
。所以最坏的情况是7美分b $ b分钟才能获得释放。

在这种情况下，我认为slice没有被标记为释放，但是在
中使用，所以它永远不会被返回到操作系统。 p>

有可能你没有足够长的时间进行GC扫描，然后进行OS返回扫描，可能会在扫描后7分钟最后的大脉搏。你可以使用 runtime.FreeOSMemory 显式强制执行，但请记住，除非GC已运行，否则它不会执行任何操作。

（编辑：请注意，您可以使用 runtime.GC（）强制垃圾回收，但显然您需要小心使用它的频率;您可能能够将它与连接中的突然下降尖峰同步）。

作为一个轻微的抛开，我找不到明确的来源（除了第二个线程I在有人提到同样的东西的时候发布），但我记得它多次提到，并非所有的内存使用都是真实的内存。如果它由运行时分配但实际上未被程序使用，则操作系统实际上已经使用了内存，无论 top 或 MemStats <























$ b $编辑：作为评论中的Kostix notex并且支持JimB的回答，这个问题被转移到了Golang坚果上，我们从Dmitri Vyukov那里得到了一个相当明确的答案：



https://groups.google.com/forum/#!topic/golang-nuts/0WSOKnHGBZE/discussion

我今天没有解决方案。
大部分内存似乎被goroutine堆栈占用，我们不会释放内存给OS。
在下一个版本中会更好一些。

所以我概述的只适用于堆变量， Goroutine堆栈将永远不会被释放。这与我的最后一个并非所有显示分配的系统内存是真实内存的相互作用究竟有何不同。

We wrote the simplest possible TCP server (with minor logging) to examine the memory footprint (see tcp-server.go below)

The server simply accepts connections and does nothing. It is being run on an Ubuntu 12.04.4 LTS server (kernel 3.2.0-61-generic) with Go version go1.3 linux/amd64.

The attached benchmarking program (pulse.go) creates, in this example, 10k connections, disconnects them after 30 seconds, repeats this cycle three times, and then continuously repeats small pulses of 1k connections/disconnections. The command used to test was ./pulse -big=10000 -bs=30.

The first attached graph is obtained by recording runtime.ReadMemStats when the number of clients has changed by a multiple of 500, and the second graph is the RES memory size seen by "top" for the server process.

The server starts with a negligible 1.6KB of memory. Then the memory is set by the "big" pulses of 10k connections at ~60MB (as seen by top), or at about 16MB "SystemMemory" as seen by ReadMemStats. As expected, when the 10K pulses end, the in-use memory drops, and eventually the program starts releasing memory back to OS as evidenced by the grey "Released Memory" line.

The problem is that the System Memory (and correspondingly, the RES memory seen by "top") never drops significantly (although it drops a little as seen in the second graph).

We would expect that after the 10K pulses end, memory would continue to be released until the RES size is the minimum needed for handling each 1k pulse (which is 8m RES as seen by "top" and 2MB in-use reported by runtime.ReadMemStats). Instead, the RES stays at about 56MB and in-use never drops from its highest value of 60MB at all.

We want to ensure scalability for irregular traffic with occasional spikes as well as be able to run multiple servers on the same box that have spikes at different times. Is there a way to effectively ensure that as much memory is released back to the system as possible in a reasonable time frame?

Code https://gist.github.com/eugene-bulkin/e8d690b4db144f468bc5 :

server.go:

package main

import (
  "net"
  "log"
  "runtime"
  "sync"
)
var m sync.Mutex
var num_clients = 0
var cycle = 0

func printMem() {
  var ms runtime.MemStats
  runtime.ReadMemStats(&ms)
  log.Printf("Cycle #%3d: %5d clients | System: %8d Inuse: %8d Released: %8d Objects: %6d\n", cycle, num_clients, ms.HeapSys, ms.HeapInuse, ms.HeapReleased, ms.HeapObjects)
}

func handleConnection(conn net.Conn) {
  //log.Println("Accepted connection:", conn.RemoteAddr())
  m.Lock()
  num_clients++
  if num_clients % 500 == 0 {
    printMem()
  }
  m.Unlock()
  buffer := make([]byte, 256)
  for {
    _, err := conn.Read(buffer)
    if err != nil {
      //log.Println("Lost connection:", conn.RemoteAddr())
      err := conn.Close()
      if err != nil {
        log.Println("Connection close error:", err)
      }
      m.Lock()
      num_clients--
      if num_clients % 500 == 0 {
        printMem()
      }
      if num_clients == 0 {
        cycle++
      }
      m.Unlock()
      break
    }
  }
}

func main() {
  printMem()
  cycle++
  listener, err := net.Listen("tcp", ":3033")
  if err != nil {
    log.Fatal("Could not listen.")
  }
  for {
    conn, err := listener.Accept()
    if err != nil {
      log.Println("Could not listen to client:", err)
      continue
    }
    go handleConnection(conn)
  }
}

pulse.go:

package main

import (
  "flag"
  "net"
  "sync"
  "log"
  "time"
)

var (
  numBig = flag.Int("big", 4000, "Number of connections in big pulse")
  bigIters = flag.Int("i", 3, "Number of iterations of big pulse")
  bigSep = flag.Int("bs", 5, "Number of seconds between big pulses")
  numSmall = flag.Int("small", 1000, "Number of connections in small pulse")
  smallSep = flag.Int("ss", 20, "Number of seconds between small pulses")
  linger = flag.Int("l", 4, "How long connections should linger before being disconnected")
)

var m sync.Mutex

var active_conns = 0
var connections = make(map[net.Conn] bool)

func pulse(n int, linger int) {
  var wg sync.WaitGroup

  log.Printf("Connecting %d client(s)...\n", n)
  for i := 0; i < n; i++ {
    wg.Add(1)
    go func() {
      m.Lock()
      defer m.Unlock()
      defer wg.Done()
      active_conns++
      conn, err := net.Dial("tcp", ":3033")
      if err != nil {
        log.Panicln("Unable to connect: ", err)
        return
      }
      connections[conn] = true
    }()
  }
  wg.Wait()
  if len(connections) != n {
    log.Fatalf("Unable to connect all %d client(s).\n", n)
  }
  log.Printf("Connected %d client(s).\n", n)
  time.Sleep(time.Duration(linger) * time.Second)
  for conn := range connections {
    active_conns--
    err := conn.Close()
    if err != nil {
      log.Panicln("Unable to close connection:", err)
      conn = nil
      continue
    }
    delete(connections, conn)
    conn = nil
  }
  if len(connections) > 0 {
    log.Fatalf("Unable to disconnect all %d client(s) [%d remain].\n", n, len(connections))
  }
  log.Printf("Disconnected %d client(s).\n", n)
}

func main() {
  flag.Parse()
  for i := 0; i < *bigIters; i++ {
    pulse(*numBig, *linger)
    time.Sleep(time.Duration(*bigSep) * time.Second)
  }
  for {
    pulse(*numSmall, *linger)
    time.Sleep(time.Duration(*smallSep) * time.Second)
  }
}

解决方案

First, note that Go, itself, doesn't always shrink its own memory space:

https://groups.google.com/forum/#!topic/Golang-Nuts/vfmd6zaRQVs

The heap is freed, you can check this using runtime.ReadMemStats(), but the processes virtual address space does not shrink -- ie, your program will not return memory to the operating system. On Unix based platforms we use a system call to tell the operating system that it can reclaim unused parts of the heap, this facility is not available on Windows platforms.

But you're not on Windows, right?

Well, this thread is less definitive, but it says:

https://groups.google.com/forum/#!topic/golang-nuts/MC2hWpuT7Xc

As I understand, memory is returned to the OS about 5 minutes after is has been marked as free by the GC. And the GC runs every two minutes top, if not triggered by an increase in memory use. So worst-case would be 7 minutes to be freed.

In this case, I think that the slice is not marked as freed, but in use, so it would never be returned to the OS.

It's possible you weren't waiting long enough for the GC sweep followed by the OS return sweep, which could be up to 7 minutes after the final "big" pulse. You can explicitly force this with runtime.FreeOSMemory, but keep in mind that it won't do anything unless the GC has been run.

(Edit: Note that you can force garbage collection with runtime.GC() though obviously you need to be careful how often you use it; you may be able to sync it with sudden downward spikes in connections).

As a slight aside, I can't find an explicit source for this (other than the second thread I posted where someone mentions the same thing), but I recall it being mentioned several times that not all of the memory Go uses is "real" memory. If it's allocated by the runtime but not actually in use by the program, the OS actually has use of the memory regardless of what top or MemStats says, so the amount of memory the program is "really" using is often very overreported.

---

Edit: As Kostix notex in the comments and supports JimB's answer, this question was crossposted on Golang-nuts and we got a rather definitive answer from Dmitri Vyukov:

https://groups.google.com/forum/#!topic/golang-nuts/0WSOKnHGBZE/discussion

I don't there is a solution today. Most of the memory seems to be occupied by goroutine stacks, and we don't release that memory to OS. It will be somewhat better in the next release.

So what I outlines only applies to heap variables, memory on a Goroutine stack will never be released. How exactly this interacts with my last "not all shown allocated system memory is 'real memory'" point remains to be seen.

这篇关于Go 1.3垃圾收集器不会将服务器内存释放回系统的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！