emarket_data_explorer.database

This module provides the database functionality.

Todo:

1. use data_source to dynamically update, not hardcoded “shopee”

2. implement write_index and read_index and List cli command in v1.5.

3. remove unused datatype and also move it to the datatype file v1.4.

4. abstract a DatabaseHandler class and rename to ShopeeDatabaseHandler in v1.4.

class emarket_data_explorer.database.DBResponseForCSV(response: DataFrame, error: int)

this class defines the format of database response for data in CSV

error: int

Alias for field number 1

response: DataFrame

Alias for field number 0

class emarket_data_explorer.database.DBResponseForIndex(response: List[Dict[str, Any]], error: int)

this class defines the format of database response for index in JSON

error: int

Alias for field number 1

response: List[Dict[str, Any]]

Alias for field number 0

class emarket_data_explorer.database.ShopeeAsyncDatabaseHandler(data_path: Path, db_path: Path)

this class provides the functions of read and write into CSV and JSON for scrapped data

write_csv(product_container: DataFrame, my_keyword: str, data_source: int, crawler_mode: str) → int

write the scrapped data stored in dataframe into CSV