Universal Pathlib

Universal Pathlib is a Python library that extends the pathlib_abc.JoinablePath API to provide a pathlib.Path-like interface for a variety of backend filesystems via filesystem_spec.

Installation

Install the latest version of universal_pathlib with pip or conda. Please note that while this will install fsspec as a dependency, for some filesystems, you have to install additional packages. For example, to use S3, you need to install s3fs, or better depend on fsspec[s3]:

PyPI

python -m pip install universal_pathlib

conda

conda install -c conda-forge universal_pathlib

Adding universal_pathlib to your project

Below is a pyproject.toml based example for adding universal_pathlib to your project as a dependency if you want to use it with s3 and http filesystems:

[project]
name = "myproject"
requires-python = ">=3.9"
dependencies = [
    "universal_pathlib>=0.3.10",
    "fsspec[s3,http]",
]

See filesystem_spec/pyproject.toml for an overview of the available fsspec extras.

Basic Usage

# pip install universal_pathlib fsspec[s3]
>>> from upath import UPath
>>>
>>> s3path = UPath("s3://test_bucket") / "example.txt"
>>> s3path.name
example.txt
>>> s3path.stem
example
>>> s3path.suffix
.txt
>>> s3path.exists()
True
>>> s3path.read_text()
'Hello World'

For more examples, see the example notebook here.

Currently supported filesystems (and protocols)

file: and local: Local filesystem
memory: Ephemeral filesystem in RAM
az:, adl:, abfs: and abfss: Azure Storage (requires adlfs)
data: RFC 2397 style data URLs (requires fsspec>=2023.12.2)
ftp: FTP filesystem
github: GitHub repository filesystem
hf: Hugging Face filesystem (requires huggingface_hub)
http: and https: HTTP(S)-based filesystem
hdfs: Hadoop distributed filesystem
gs: and gcs: Google Cloud Storage (requires gcsfs)
s3: and s3a: AWS S3 (requires s3fs to be installed)
sftp: and ssh: SFTP and SSH filesystems (requires paramiko)
smb: SMB filesystems (requires smbprotocol)
webdav, webdav+http: and webdav+https: WebDAV-based filesystem on top of HTTP(S) (requires webdav4[fsspec])

It is likely, that other fsspec-compatible filesystems are supported through the default implementation. But because they are not tested in the universal_pathlib test-suite, correct behavior is not guaranteed. If you encounter any issues with a specific filesystem using the default implementation, please open an issue. We are happy to add support for other filesystems via custom UPath implementations. And of course, contributions for new filesystems are welcome!

Class hierarchy

The class hierarchy for UPath implementations and their relation to base classes in pathlib_abc and the stdlib pathlib classes are visualized in the following diagram. Please be aware that the pathlib_abc.JoinablePath, pathlib_abc.ReadablePath, and pathlib_abc.WritablePath classes are currently not actual parent classes of the stdlib pathlib classes. This might occur in later Python releases, but for now, the mental model you should keep is represented by the diagram.

flowchart TB

  subgraph p0[pathlib_abc]
    X ----> Y
    X ----> Z
  end

  subgraph s0[pathlib]
    X -.-> A

    A----> B
    A--> AP
    A--> AW

    Y -.-> B
    Z -.-> B

    B--> BP
    AP----> BP
    B--> BW
    AW----> BW
  end
  subgraph s1[upath]
    Y ---> U
    Z ---> U

    U --> UP
    U --> UW
    BP ---> UP
    BW ---> UW
    U --> UL
    U --> US3
    U --> UH
    U -.-> UO
  end

  X(JoinablePath)
  Y(WritablePath)
  Z(ReadablePath)

  A(PurePath)
  AP(PurePosixPath)
  AW(PureWindowsPath)
  B(Path)
  BP(PosixPath)
  BW(WindowsPath)

  U(UPath)
  UP(PosixUPath)
  UW(WindowsUPath)
  UL(FilePath)
  US3(S3Path)
  UH(HttpPath)
  UO(...Path)

  classDef na fill:#f7f7f7,stroke:#02a822,stroke-width:2px,color:#333
  classDef np fill:#f7f7f7,stroke:#2166ac,stroke-width:2px,color:#333
  classDef nu fill:#f7f7f7,stroke:#b2182b,stroke-width:2px,color:#333

  class X,Y,Z na
  class A,AP,AW,B,BP,BW,UP,UW np
  class U,UL,US3,UH,UO nu

  style UO stroke-dasharray: 3 3

  style p0 fill:none,stroke:#0a2,stroke-width:3px,stroke-dasharray:3,color:#0a2
  style s0 fill:none,stroke:#07b,stroke-width:3px,stroke-dasharray:3,color:#07b
  style s1 fill:none,stroke:#d02,stroke-width:3px,stroke-dasharray:3,color:#d02

To be concrete this currently means:

# for all supported Python versions:
from pathlib import Path
from upath import UPath
from upath.types import JoinablePath

assert isinstance(Path(), JoinablePath) is False
assert isinstance(UPath(), JoinablePath) is True

When instantiating UPath the returned instance type is determined by the path, or better said, the "protocol" that was provided to the constructor. The UPath class will return a registered implementation for the protocol, if available. If no specialized implementation can be found but the protocol is available through fsspec, it will return a UPath instance and provide filesystem access with a default implementation. Please note the default implementation can not guarantee correct behavior for filesystems that are not tested in the test-suite.

Local paths and url paths

If a local path is without protocol is provided UPath will return a PosixUPath or WindowsUPath instance. These two implementations are 100% compatible with the PosixPath and WindowsPath classes of their specific Python version. They're tested against a large subset of the CPython pathlib test-suite to ensure compatibility.

If a local urlpath is provided, i.e. a "file://" or "local://" URI, the returned instance type will be a FilePath instance. This class is a subclass of UPath that provides file access via LocalFileSystem from fsspec. You can use it to ensure that all your local file access is done through fsspec as well.

All local UPath types are os.PathLike, but only the PosixUPath and WindowsUPath are subclasses of pathlib.Path.

UPath public class API

The public class interface of UPath extends pathlib.Path via attributes that simplify interaction with filesystem_spec. Think of the UPath class in terms of the following code:

from pathlib_abc import ReadablePath, WritablePath
from typing import Any, Mapping
from fsspec import AbstractFileSystem

class UPath(ReadablePath, WriteablePath):
    # the real implementation is more complex, but this is the general idea

    @property
    def protocol(self) -> str:
        """The fsspec protocol for the path."""

    @property
    def storage_options(self) -> Mapping[str, Any]:
        """The fsspec storage options for the path."""

    @property
    def path(self) -> str:
        """The path that a fsspec filesystem can use."""

    @property
    def fs(self) -> AbstractFileSystem:
        """The cached fsspec filesystem instance for the path."""

These attributes are used to provide a public interface to move from the UPath instance to more fsspec specific code:

from upath import UPath
from fsspec import filesystem

p = UPath("s3://bucket/file.txt", anon=True)

fs = filesystem(p.protocol, **p.storage_options)  # equivalent to p.fs

with fs.open(p.path) as f:
    data = f.read()

Supported Interface

Universal Pathlib provides an implementation of the pathlib.Path interface across different Python versions. The following table shows the compatibility matrix for stdlib pathlib.Path attributes and methods. Methods supported in UPath should correctly work for all supported Python versions. If not, we consider it a bug.

Method/Attribute	py3.9	py3.10	py3.11	py3.12	py3.13	py3.14	UPath
Pure Paths
Operators
`__truediv__`	✅	✅	✅	✅	✅	✅	✅
`__rtruediv__`	✅	✅	✅	✅	✅	✅	✅
Access individual Parts
`parts`

universal-pathlib

Description