ZFS Log Compression

Again and again I was annoyed when I found logs bz compressed on FreeBSD systems with ZFS root. Would it not be more effective and efficient to let ZFS do the compression transparently with zstd?

Today I got fed up and searched what others thought of this and I found this Article Log Compressin on FreeBSD.

They share my sentiment. Their remedy is to make sure zfs compression is enabled and switch off newsyslog bz compression:

Since we want ZFS to do the compression, remove the J from this column for each such log file listed there. Once finished with the changes, save and exit newsyslog.conf.

Doing the actual work

First we need to find every relevant file and then edit them all. Doing this by hand would be annoying on a single machine. What about hosts with many jails?

The relevant files

The relevant files are /etc/newsyslog.conf, all files /etc/newsyslog.conf.d/*.conf and /usr/local/etc/newsyslog.conf.d/*.conf.

So use sed

My first approach was to 

sed s/J// /etc/newsyslog.conf /etc/newsyslog.conf.d/*.conf

But this broke newsyslog and it refused to start with a message about / not being a valid flag.

Find where the error occured: 

newsyslog -vn

/etc/newsyslog.conf.d/opensm.conf is the culprit. It has the path to a pid-file following the J.

So if something follows the flag J, it needs to be replaced by -, not deleted. So find a J between two tabs and replace it by a - between two tabs. \t stands for tab. 

sed 's/\tJ\t/\t-\t/'

Sed allows multiple commands, seperated by ;.

This works: 

sed 's/\tJ\t/\t-\t/;s/J//' /etc/newsyslog.conf /etc/newsyslog.conf.d/*.conf

The replacement of J by - has to come first. If we delete first there is no J left to replace.

Be thorough

Tabs or whitespaces

There are some lines in the config files that end on a J with different numbers in spaces in front. I do not want to leave behind a file that has whitespace at the end of lines. 

sed 's/[\t ]*J//'

Comments

Removing Js from comments will not break anything, but it is poor practice. So only work on lines not starting with #. 

sed '/^[^#]/s/J//'

All together

  1. On /etc/newsyslog.conf and all *.conf files in the newsyslog.conf.d directories.
  2. Work only on none comment lines.
  3. Remove trailing Js with preceeding whitespace.
  4. Replace single Js with -.
  5. Remove the remaining Js.

One-Liners

Here two long, ugly one-liners for future me to cut and paste.

For a host

sed -i '' '/^[^#]/s/[\t ]*J$//;/^[^#]/s/\tJ\t/\t-\t/;/^[^#]/s/J//' etc/newsyslog.conf etc/newsyslog.conf.d/*.conf /etc/usr/local/etc/newsyslog.conf.d/*.conf

Loop over jails

for JAIL in $(jls name); do sudo sed -i '' '/^[^#]/s/[\t ]*J$//;/^[^#]/s/\tJ\t/\t-\t/;/^[^#]/s/J//' /jail/$JAIL/etc/newsyslog.conf /jail/$JAIL/etc/newsyslog.conf.d/*.conf /jail/$JAIL/usr/local/etc/newsyslog.conf.d/*.conf ; done

for JAIL in $(jls name); do sudo service -j $JAIL newsyslog restart ; done