19

After force running logrotate, my app keeps writing to my_app.log.1 (the old log that should be archived later) instead of my_app.log.

This make my_app.log an empty file, therefore logrotate runs without any effect. And my_app.log.1 keep growing to gigabytes.

I am running Ubuntu 12.04. My app is a Node.js app using pm2. Following is my logrotate configuration:

"/var/log/my_app/*.log" {
  daily
  size 50M
  rotate 10
  missingok
  compress
  delaycompress
  notifempty
}

I know I put notifempty there, but why is my_app.log.1 written to in the first place?

Tung Nguyen
  • 1,874
  • 3
  • 18
  • 28
  • 2
    If the application keeps the log file open while it is running, it is not affected by the file name being changed. It continues to write in the file. `logrotate` can be configured (in a `postrotate` script) to restart the application (or send it a `HUP` signal, if the application responds to it) after it renames the files. There are examples in the documentation. – axiac Aug 16 '17 at 12:43

1 Answers1

34

I finally figured out how to solve the problem.

This was because the log file was being written by pm2. logrotate changed its name to my_app.log.1 and created new my_app.log file, but pm2 did not care about this and kept writing to my_app.log.1.

I solved the problem by replacing the notifempty option by copytruncate and then restarted pm2. After fixing, notifempty can be added back but I do not really need it.

See the logrotate reference for more information. Hope this will help other folks getting a similar problem.

gdvalderrama
  • 713
  • 1
  • 17
  • 26
Tung Nguyen
  • 1,874
  • 3
  • 18
  • 28
  • 2
    There is a small problem with copytruncate. While the file is being copied and truncated, some amount of data will be lost. How do you solve this? – Reyansh Kharga May 27 '21 at 08:46
  • 2
    I think a better solution is to use postrotate script to restart the pm2 processes but adds additional complexity. – Reyansh Kharga May 27 '21 at 08:47