<!-- markdownlint-disable MD026 -->

Concurrent Log Handler (CLH)

The concurrent-log-handler package provides robust logging handlers for Python's standard logging package (PEP 282). It enables multiple processes (and threads) to safely write to a single log file, with built-in support for size-based and time-based log rotation and optional log compression.

This package is meant for applications that run in multiple processes, potentially across different hosts sharing a network drive, and require a centralized logging solution without the complexity of external logging services.

NOTE: If you're reading this and the links to files like CHANGELOG.md don't work, or links within the document either, try viewing this document on GitHub instead.

What's new

See CHANGELOG.md for details.

• Version 0.9.29: (February 2026)

  • Fix race conditions when a handler created before fork() is used by multiple child processes. Child processes that inherit a handler now automatically reopen the lock file for independent lock isolation, and an in-process threading lock prevents concurrent threads from bypassing flock() serialization.
    • Re-initializing logging post-fork (see Usage Guidelines) is still recommended.
  • Depend on portalocker >= 2.6.0 instead of 1.6.0. Earlier versions of portalocker on Windows can be problematic.
  • Add finalize_handler_configuration() hook to ConcurrentTimedRotatingFileHandler to allow customization of the handler before the first rollover.

• Version 0.9.28: (June 10th, 2025)

• Fixes errors when apps, esp. asyncio based, try to log during interpreter shutdown. Issue #80

• Fix missing rollovers when a worker was restarted before the next logging event, but after the last scheduled rollover. Issue #81

• Version 0.9.27: (June 6th, 2025)

  • Fixes Issue #73 Fix timed rotation handler's file cleanup logic.
  • Fixes Issue #79 Harden timed handler's rollover mechanism against timestamp errors or other sync corruption.

• Important Notice (June 2025): Background Logging Utility Removed

  • The concurrent_log_handler.queue module was removed in version 0.9.28.
  • It has compatibility issues with complex logging setups and other robustness concerns.
  • Recommendation:
    • Simply use the standard CLH handlers (ConcurrentRotatingFileHandler or ConcurrentTimedRotatingFileHandler) directly in your application. Synchronous logging calls are simpler, more reliable, and performant enough for most use cases.
    • If you need non-blocking logging calls, use the standard library patterns shown in Performance Patterns.
  • The core CLH handlers remain fully supported and are not affected by this deprecation.

• Version 0.9.26: (May 2025)

  • Improved performance, especially on POSIX systems.
  • Added testing for Python 3.13 and improved project configuration and documentation.

Key Features

• Concurrent Logging: Multiple processes and threads safely write to the same log file.
• File Rotation: Size-based and time-based rotation with optional compression.
• Cross-Platform: Windows and POSIX support with reliable file locking.
• Customizable: Control naming, permissions, line endings, and lock file placement.
• Performance Optimized: Keeps files open between writes for better performance.
• Python 3.6 through current versions: Modern Python support.
• Focused Design: Reliably handles file operations. For non-blocking behavior, see our recommended Application-Level Performance Patterns, including patterns for graceful degradation to synchronous logging, or using sync logging only for higher priority levels.

Primary Use Cases

CLH is primarily designed for scenarios where:

• Multiple processes of a Python application need to log to a shared file.
• These processes might be on the same machine or on different machines accessing a shared network drive.
• Log files need to be automatically rotated based on size or time.

Note that this package is not primarily intended for intensive high-throughput logging scenarios, but rather for general-purpose logging in multi-process applications.

Alternatives to CLH

While CLH offers a robust file-based solution, consider these alternatives for different needs:

• Cloud Logging Services: Azure Monitor, AWS CloudWatch Logs, Google Cloud Logging, Logstash, etc. These are excellent for distributed systems and offer advanced analysis features.
• Custom Logging Server: Implement a centralized logging server as demonstrated in the Python Logging Cookbook. CLH's QueueHandler and QueueListener can be adapted for this pattern.

Installation

Install the package using pip:

pip install concurrent-log-handler

This will also install portalocker. On Windows, portalocker has a dependency on pywin32. Note that we now require at least portalocker 2.6.0, which among other things, explicitly declares its dependency on pywin32 if needed.

To install from source:

python setup.py install

Quick Start: Basic Usage

Here's a simple example using ConcurrentRotatingFileHandler for size-based rotation:

import logging
from concurrent_log_handler import ConcurrentRotatingFileHandler
import os

logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)

# Use an absolute path for the log file
logfile = os.path.abspath("mylogfile.log")

# Configure the handler: rotate after 512KB, keep 5 backups
# Mode "a" for append
rotate_handler = ConcurrentRotatingFileHandler(
    logfile, "a", maxBytes=512 * 1024, backupCount=5
)
logger.addHandler(rotate_handler)

logger.info("This is an exciting log message!")
logger.info("Multiple processes can write here concurrently.")

For a few more basic code examples, see src/example.py. For more advanced usage including non-blocking patterns, see the Performance Patterns guide.

Important Usage Guidelines

To ensure correct and reliable operation when using CLH in a multi-process environment, please keep the following in mind:

1. Handler Instantiation per Process:

   • Each process must create its own instance of the CLH handler (ConcurrentRotatingFileHandler or ConcurrentTimedRotatingFileHandler).
   • You cannot serialize a handler instance and reuse it in another process.
     • For example, you cannot pass it from a parent to a child process via multiprocessing in spawn mode.
   • This limitation is because the file lock objects and other internal states within the handler cannot be safely serialized and shared across process boundaries.
   • This requirement does not apply to threads within the same process; threads can share a single CLH instance.
   • This requirement also applies to child processes created via fork() (e.g., Gunicorn with --preload, uWSGI, or multiprocessing with the fork start method). Forked children inherit the parent's file descriptors, which causes file locks to silently fail to serialize between processes. See item 3 below for the recommended pattern.

2. Multiprocessing and Spawn mode:

   Just to reemphasize the point above:

   • If you use multiprocessing with the default spawn start method (the default on Windows and macOS), each child process must create its own CLH handler instance.
   • In your child process startup code, instantiate the handler as shown in the example above.
   • Usually this means you can call your standard logging setup function in the child.
   • Don't just initialize your logging code in the parent process and allow child processes to inherit loggers.

3. Using fork() (Gunicorn, uWSGI, multiprocessing with fork start method):

On Linux, depending on the version of Python, multiprocessing defaults to the fork start method. Frameworks like Gunicorn with --preload and uWSGI also fork worker processes from a parent that has already initialized logging.

While CLH v0.9.29+ has in