Skip to main content
Version: devel

common.storages.configuration

SchemaStorageConfiguration Objects

@configspec
class SchemaStorageConfiguration(BaseConfiguration)

[view_source]

schema_volume_path

path to volume with default schemas

import_schema_path

path from which to import a schema into storage

export_schema_path

path to which export schema from storage

external_schema_format

format in which to expect external schema

NormalizeStorageConfiguration Objects

@configspec
class NormalizeStorageConfiguration(BaseConfiguration)

[view_source]

normalize_volume_path

path to volume where normalized loader files will be stored

make_fsspec_url

def make_fsspec_url(scheme: str, fs_path: str, bucket_url: str) -> str

[view_source]

Creates url from fs_path and scheme using bucket_url as an url template

Arguments:

  • scheme str - scheme of the resulting url
  • fs_path str - kind of absolute path that fsspec uses to locate resources for particular filesystem.
  • bucket_url str - an url template. the structure of url will be preserved if possible

FilesystemConfiguration Objects

@configspec
class FilesystemConfiguration(BaseConfiguration)

[view_source]

A configuration defining filesystem location and access credentials.

When configuration is resolved, bucket_url is used to extract a protocol and request corresponding credentials class.

  • s3
  • gs, gcs
  • az, abfs, adl, abfss, azure
  • file, memory
  • gdrive
  • sftp

read_only

Indicates read only filesystem access. Will enable caching

max_state_files

Maximum number of pipeline state files to keep; 0 or negative value disables cleanup.

protocol

@property
def protocol() -> str

[view_source]

bucket_url protocol

fingerprint

def fingerprint() -> str

[view_source]

Returns a fingerprint of bucket schema and netloc.

Returns:

  • str - Fingerprint.

make_url

def make_url(fs_path: str) -> str

[view_source]

Makes a full url (with scheme) form fs_path which is kind-of absolute path used by fsspec to identify resources. This method will use bucket_url to infer the original form of the url.

__str__

def __str__() -> str

[view_source]

Return displayable destination location

is_local_path

@staticmethod
def is_local_path(url: str) -> bool

[view_source]

Checks if url is a local path, without a schema

make_local_path

@staticmethod
def make_local_path(file_url: str) -> str

[view_source]

Gets a valid local filesystem path from file:// scheme. Supports POSIX/Windows/UNC paths

Returns:

  • str - local filesystem path

make_file_url

@staticmethod
def make_file_url(local_path: str) -> str

[view_source]

Creates a normalized file:// url from a local path

netloc is never set. UNC paths are represented as file://host/path

This demo works on codespaces. Codespaces is a development environment available for free to anyone with a Github account. You'll be asked to fork the demo repository and from there the README guides you with further steps.
The demo uses the Continue VSCode extension.

Off to codespaces!

DHelp

Ask a question

Welcome to "Codex Central", your next-gen help center, driven by OpenAI's GPT-4 model. It's more than just a forum or a FAQ hub – it's a dynamic knowledge base where coders can find AI-assisted solutions to their pressing problems. With GPT-4's powerful comprehension and predictive abilities, Codex Central provides instantaneous issue resolution, insightful debugging, and personalized guidance. Get your code running smoothly with the unparalleled support at Codex Central - coding help reimagined with AI prowess.