Python ate my log message

0

When trying out a modification to a Python program, I noticed a unfamiliar ingredient:
among the messages that the program must had been sending to the syslog
were lacking.

Right here’s a program easy-daemon.py with only ample code to
highlight the placement:

#!/usr/bin/env python3
# Requires the following third-occasion Python programs:
# python-daemon: https://pypi.python.org/pypi/python-daemon
# pylockfile: https://github.com/openstack/pylockfile
import logging.handlers
import sys
import daemon
import daemon.pidfile as pidlockfile
import daemon.runner
from lockfile import AlreadyLocked
from lockfile import LockTimeout


def fundamental(): 
    pidpath = "/tmp/easy-daemon.pid"
    logger = logging.getLogger("easy-daemon")
    logger.setLevel(logging.INFO)
    formatter = logging.Formatter(
        "%(asctime)s %(module)s[%(process)d] %(levelname)s: %(message)s",
        "%Y-%m-%d %H: %M: %S")
    handler = logging.handlers.SysLogHandler(deal with="/dev/log")
    handler.setFormatter(formatter)
    logger.addHandler(handler)

    logger.recordsdata("Initiating.")
    # Win pidfile
    pidf = pidlockfile.TimeoutPIDLockFile(pidpath, 10)

    # Rob away any stale PID recordsdata, left gradual by previous invocations
    if daemon.runner.is_pidfile_stale(pidf): 
        logger.warning("Weeding out stale PID lock file %s", pidf.route)
        pidf.break_lock()

    daemon_context = daemon.DaemonContext(pidfile=pidf, umask=0o22)
    strive: 
        daemon_context.originate()
    with the exception of (AlreadyLocked, LockTimeout): 
        logger.significant("Didn't lock pidfile %s", pidpath)
        sys.exit(1)

    logger.recordsdata("Daemonized.")
    logger.recordsdata("Yet any other message.")

if __name__ == "__main__": 
    fundamental()

To reproduce the worm, go the program and grep the machine logs for the
string “easy-daemon”. You’ll watch “Initiating.” and “Yet any other Message.” but no longer
a “Daemonized.” message.

I tried this on a few distributions. I will also no longer reproduce the placement on Debian
Stretch, but did reproduce it on CentOS, Arch Linux, and Debian Buster. I
expected to search out the message “Daemonized.” On the reproducing programs, the entirely
string within the logs used to be:

Aug 28 16: 30: 16 computer 2020-08-28 16: 30: 16 easy-daemon[26333] INFO: Initiating.
Aug 28 16: 30: 16 computer 2020-08-28 16: 30: 16 easy-daemon[26335] INFO: Yet any other message.

Operating strace on the program and shopping for the syscalls that accessed
the syslog, I stumbled on:

socket(AF_UNIX, SOCK_DGRAM|SOCK_CLOEXEC, 0) = 3
connect(3, {sa_family=AF_UNIX, sun_path="/dev/log"}, 10) = 0
# ... and sometime later:
sendto(3, "<14>2020-08-27 06: 16: 45 easy-daemon[109680] INFO: Daemonized.", 64, 0, NULL, 0) = -1 EBADF (Sinister file descriptor)
end(3)                                = -1 EBADF (Sinister file descriptor)

Why is this program making an try to jot down to after which end an invalid file
descriptor? The acknowledge becomes a runt extra evident with an realizing
of what the python-daemon package does and a study of
logging.handlers.SysLogHandler.

By default, python-daemon closes all originate file descriptors. That capabilities
file descriptors referenced by Python objects. The file descriptor for
the syslog is referenced by the socket member of SysLogHandler objects.
The principle write to the syslog wrote data to an already-closed file descriptor,
which resulted in a “end socket and reconnect” sequence.

I first saw this situation in Python 2.7, on CentOS 7. Right here the strace
output is noteworthy extra outlandish:

sendto(3, "<14>2020-08-27 06: 43: 12 easy-daemon[3708] INFO: Daemonized.", 62, 0, NULL, 0) = -1 EBADF (Sinister file descriptor)
socket(AF_UNIX, SOCK_DGRAM, 0)          = 3
end(3)                                = 0
connect(3, {sa_family=AF_UNIX, sun_path="/dev/log"}, 10) = -1 EBADF (Sinister file descriptor)
end(3)                                = -1 EBADF (Sinister file descriptor)
write(2, "Traceback (newest call final):n", 35) = 35

Right here, after a failed write, SysLogHandler today calls the socket
machine call with out closing the faded socket. It then closes the socket it gorgeous
created and tries to connect to the closed socket! This is a confluence
of bugs: the dangling file descriptor worm I’ve been discussing, and
Python Teach 17981 (“SysLogHandler
closes connection sooner than utilizing it”).

Right here’s the inoffensive but problematic line from
logging.handlers.SysLogHandler._connect_unixsocket:

self.socket = socket.socket(socket.AF_UNIX, use_socktype)

That line creates a socket and assigns it to self.socket. As a
aspect-attain, it causes the faded self.socket to be unreferenced and readily available
for rubbish sequence. When the faded self.socket is rubbish-restful,
its .end components gets called, because Python doesn’t know that it’s miles
already closed. Unluckily, each the faded and novel socket object veteran file
descriptor 3, so the novel object now references a closed file descriptor.

The resolution to Python Teach 17981 is to end the socket on error.
The article’s .end components will be called sooner than increasing the novel socket,
somewhat than after. The fix used to be committed upstream in Could per chance of 2013, but
it has no longer yet made its technique to CentOS 7. Teach 17981 is a purple herring
even though, and the correct situation is objects with dangling references to file
descriptors.

The easy fix in this sample program is to present an rationalization for DaemonContext
no longer to end the syslog file descriptor:

daemon_context = daemon.DaemonContext(
    pidfile=pidf,
    umask=0o22,
    files_preserve=[handler.socket],
)

One have to deem fastidiously sooner than utilizing capabilities like os.end. This
syscall works on the underlying file descriptor, which may per chance well per chance be
connected to a File object that expects to manage it’s lifetime.

Read More

Leave A Reply

Your email address will not be published.