Data Reading Functions

Reader

Recommended: A class for reading ETL files from the sync-output directory. This is the recommended way to read ETL files as it provides consistent error handling and metadata access.

Installation

from gluestick.reader import Reader

Basic Usage

# Initialize with default directories
reader = Reader()  # Uses ROOT_DIR/sync-output

# Get available streams
print(reader)  # Shows list of available streams

Key Methods

---

get(stream, default=None, catalog_types=False, **kwargs)

Reads the selected file into a pandas DataFrame.

# Basic reading
df = reader.get("users")

# Using catalog types
df = reader.get("users", catalog_types=True)

get_metadata(stream)

Retrieves metadata from parquet files.

metadata = reader.get_metadata("users")
# Returns dict of metadata key-value pairs

---

get_pk(stream)

Gets primary key(s) from parquet file metadata.

primary_keys = reader.get_pk("users")
# Returns list of primary key column names

---

get_catalog_schema

Retrieves Singer schema from catalog file.

Usage

from gluestick.singer import get_catalog_schema, to_singer


# Get schema for specific stream
schema = get_catalog_schema("users")

# Use schema with Singer export
to_singer(df, "users", OUTPUT_DIR, schema=schema)

Returns

Dictionary containing the stream’s schema definition

Notes

• Requires catalog.json in root directory
• Raises exception if stream not found in catalog
• Filters schema to include only type and properties
• Ensures array types have items dictionary

Common Patterns

Iterating through multiple streams

etl.py

import gluestick as gs

# iterate through each stream in the input directory
reader = gs.Reader()
for key in eval(str(reader)):

    # define a dataframe and apply transformations
    input_df = reader.get(key, catalog_types=True)
    input_df["tenant"] = TENANT_ID

    # define the primary keys (assuming the data is Parquet)
    key_properties = reader.get_pk(key)

    # write the data out to the output directory
    gs.to_export(input_df, key, OUTPUT_DIR, keys=key_properties)