golang memory usage keeps increasing and not decreasing (memory leak) | Discord Gophers | Page 1

winged junco May 28, 2024, 7:31 AM

#

The details are within this task, I am asking for your help on an important issue that I cannot find a solution to.

https://stackoverflow.com/questions/78542496/golang-memory-usage-keeps-increasing-and-not-decreasing-memory-leak

Stack Overflow

golang memory usage keeps increasing and not decreasing (memory leak)

In my project I created with golang gin, my memory usage is constantly increasing as shown in the graph below, and the last thing is the container restart. This happens within a day. Even though I

fickle pagoda May 28, 2024, 10:57 AM

#

@winged junco - This would more info regarding source code, and debugging info that you tried. By just seeing this, it seems accurate for container to get restart when 100% memory usage.

winged junco May 28, 2024, 11:00 AM

#

@fickle pagoda I have a server side project that works with Golang gin framework. I provide database connections with Gorm (mysql). What information or codes do you need to determine the cause?

loud mauve May 28, 2024, 11:05 AM

#

honestly it looks like you're not closing some resource on the MIME side, or on the bufio reader/writer side. if you post the code of how you're handling requests that's probably the culprit

light geyser May 28, 2024, 11:16 AM

#

winged junco <@1230760892103135242> I have a server side project that works with Golang gin f...

Are you persisting something in memory?! Simply establishing a database connection would not lead to this. As mentioned by others, if possible, please provide the relevant source code.

My guess is you may be reading a lot of data at once. Consider batching. (Need source code tho to confirm bongo )

winged junco May 28, 2024, 11:30 AM

#

@loud mauve @light geyser Thank you for your help and answers, unfortunately I cannot share the repo since it is a private project. But if you add some links and source codes below, I hope it will be understandable, if more detailed is needed, I can provide them.

#

Database connection (mysql)

func ConnectMySQL() (*gorm.DB, error) {
dsn := fmt.Sprintf("%s:%s@tcp(%s:%v)/%s?charset=utf8mb4&parseTime=True&loc=Local", os.Getenv("DB_USER"), os.Getenv("DB_PASSWORD"), os.Getenv("DB_HOST"), os.Getenv("DB_PORT"), os.Getenv("DB_DATABASE"))

db, err := gorm.Open(mysql.Open(dsn), &gorm.Config{
    TranslateError: true,
    Logger:         logger.Default.LogMode(logger.Silent),
    NamingStrategy: schema.NamingStrategy{
        SingularTable: true,
    },
})

if err != nil {
    log.Printf("Failed to connect to MySQL: %v", err)
    return nil, err
}

return db, nil

}

#

Database connection (mongo)

func ConnectMongoDB() (mongo.Database, error) {
ctx, cancel := context.WithTimeout(context.Background(), 10time.Second)
defer cancel()
uri := os.Getenv("MONGO_URI")
clientOptions := options.Client().ApplyURI(uri)
c, err := mongo.Connect(ctx, clientOptions)
if err != nil {
return nil, err
}

if err := c.Ping(ctx, nil); err != nil {
    return nil, err
}

return c.Database("ClusterName"), nil

}

#

Code of how I handle requests;

I use modular structure. I have a folder consisting of repository, service and handler layers.

For example, I have an API where files of base64 type are received and added to the server. Here we define a gin.routerGroup (in the handler)

router.POST("/test-api", h.Index)
I define it as

In my index function;

func (h *handler) Index(c *gin.Context) {
var body requestBody
if err := c.ShouldBindJSON(&body); err != nil {
c.JSON(http.StatusBadRequest, utils.ErrorResponse(err, http.StatusBadRequest, "Bad Request"))
return
}

validate := validator.New()
if err := validate.Struct(body); err != nil {
    c.JSON(http.StatusBadRequest, utils.ErrorResponse(err, http.StatusBadRequest, "Bad Request"))
    return
}

..process

c.JSON(http.StatusOK, utils.SuccessResponse(response))

}

For example, it happens like this.

I don't know if it's explanatory, but this is the general structure, I can give more details.

loud mauve May 28, 2024, 11:33 AM

#

are you reusing your opened database and mongo connections? or are you calling those open functions inside your gin handlers?

#

because if you're opening a new connection to mongo and your database everytime you handle a connection that is absolutely the memory leak

#

you should re-use them

winged junco May 28, 2024, 11:33 AM

#

I continue to use it via gin handler

loud mauve May 28, 2024, 11:33 AM

#

how so? can you post that code?

winged junco May 28, 2024, 11:34 AM

#

Of course I forward it

#

I send the mysql and mongo connections created in the routes.go file to the repositories as in the code below. ; chatRepository := chat.NewRepository(
db,
mongoDB,)
chatService := chat.NewService(chatRepository)
chat.NewHandler(
router,
api,,
) ;

#

Here is an example function

#

func (r *repository) CreateChat(chat *SendChatRequest) (*SendChatRequest, error) {
go func() {
..r.collection.Collection("messages")....)}

loud mauve May 28, 2024, 11:39 AM

#

hmm okay so it's probably not your db or mongo then, are you properly closing your resources you're using inside createchat functions?

#

show me your reading/writing of bodys to handle the requests

winged junco May 28, 2024, 11:40 AM

#

Let me give you a simpler function to understand the working logic of the functions.

#

func (r *repository) GetChatByID(id string) (*ChatRequest, error) {
var chat ChatRequest
if err := r.collection.
Collection("user_chats..").
FindOne(context.Background(), bson.M{"_id": id}).
Decode(&chat); err != nil {
log.Printf("An error occurred while getting chat: %v", err)
return nil, err
}

return &chat, nil

}

#

For example, here I am pulling a value from the desired id from Mongo db. My functions work like this on both the mysql side and the mongodb side. A request is received, the necessary operations are performed, the data is received and sent as a response.

#

plain rest api logic actually

winged junco May 28, 2024, 12:58 PM

#

@loud mauve do you have any comments ?

loud mauve May 28, 2024, 8:10 PM

#

yeah not enough information here to help you debug, unless you're going to post more code that pertains to request handling and reading/writing bodies then I can't help you further

#

everything you've shown so far seems fine, but obviously you have an insidious memory leak

winged junco May 29, 2024, 7:02 AM

#

Thank you for your help, have you examined the pprof? Can we draw a conclusion from there? @loud mauve

loud mauve May 29, 2024, 7:02 AM

#

yes, and it points to your handling of MIME/reading/writing bodies

winged junco May 29, 2024, 7:09 AM

#

@loud mauve thanks for the help, well for example runtime.GC() every hour
What would be the solution if I run it and clear the memory? My goal is to somehow clean the memory without restarting the container, .GC() does not break sockets or anything, but the document says it blocks incoming transactions at the same time. Do you have any information about what would happen if I applied a similar solution?

loud mauve May 29, 2024, 7:12 AM

#

this shouldn't be happening at all, so that solution, while it may work is a bandaid on a gunshot wound. can you repost your pprof results? the stack overflow link is no longer valid

#

you can try to read this article to see if this method helps point you to the culprit in your code: https://www.nylas.com/blog/finding-memory-leak-in-go-service-dev/

Nylas

Finding Memory Leak in Go Service

Here at Nylas, we have a process to diagnose memory leak for Go services. Tools such as pprof and minikube can help us finding the root cause.

winged junco May 29, 2024, 7:19 AM

#

@loud mauve
pprof like this, I am examining the link you sent.

loud mauve May 29, 2024, 7:19 AM

#

how long was the profiling period for that ^ graphic?

winged junco May 29, 2024, 7:23 AM

#

a few seconds

loud mauve May 29, 2024, 7:30 AM

#

yeah that's a gnarly memory leak then

#golang memory usage keeps increasing and not decreasing (memory leak)