smartcsv - GitLite

KO | EN

gitlite — search

#python #llm #go #machine-learning #deep-learning #linux #docker #nodejs #java #windows #kubernetes #vue #openai #open-source

smartcsv

(★ 368)

CSVs are awesome, yet they're pretty dumb. Let's get them smarter!

File Explorer

Download Latest Version (.zip)

- __init__.py
- exceptions.py
- reader.py
- - __init__.py
- - __init__.py
  - test_column_choices.py
  - test_column_default.py
  - test_column_validation.py
  - test_column_value_transformation.py
  - test_required_column.py
  - test_skip_column.py
- __init__.py
- base.py
- config.py
- test_csv_failures.py
- test_csv_skip_lines.py
- test_empty_rows.py
- test_high_level_invalid_csv.py
- test_iterator_interface.py
- test_model_definitions.py
- test_regressions.py
- test_valid_csv.py
- test_value_transformations.py
.gitignore
.travis.yml
LICENSE
README.md
requirements-dev.txt
setup.cfg
setup.py
tox.ini

# Installation Guide

1. Get the code

git clone https://github.com/santiagobasulto/smartcsv

Downloads the entire project code from GitHub to your computer.

cd smartcsv

Moves into the project folder you just downloaded.

2. Official Install Script

Easy Recommended

Prerequisites

Python 3 Python is required to use pip.

pip install smartcsv

Installs the package published on PyPI directly — no need to clone the source.

✅ After installing, open a new terminal and run the program's version command (e.g. --version) to confirm it worked.

Pulled directly from this repo's README.

3. Python

Easy

Prerequisites

Git Needed to download the project code from GitHub.
Python 3 On Windows, be sure to check 'Add Python to PATH' during installation.

pip install smartcsv

Installs the package published on PyPI directly — no need to clone the source.

✅ If it runs without errors and prints output in the terminal, it worked.

Pulled directly from this repo's README.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

![Travis status](https://travis-ci.org/santiagobasulto/smartcsv.svg) &nbsp; ![PyPi version](https://badge.fury.io/py/smartcsv.png)  &nbsp; ![PyPi downloads](http://img.shields.io/pypi/dm/smartcsv.svg)

# Smart and awesome CSV utils

**CSVs are awesome, yet they're pretty dumb. Let's get them smarter!**

smartcsv is a python utility to read and parse CSVs based on model definitions. Instead of just parsing the CSV into lists (like the builtin `csv` module) it adds the ability to specify models with attributes names. On top of that it adds nice features like **validation, custom parsing, failure control and nice error messages**.

```python
>>> reader = smartcsv.reader(file_object, columns=COLUMNS, fail_fast=False)
>>> my_object = next(reader)
>>> my_object['title']  # Accessed by model name.
'iPhone 5c Blue'
>>> my_object['price']  # Value transform included
Decimal("799.99")
>>> my_object['currency']  # Based on choices = ['USD', 'YEN']
'USD'
>>> my_object['url']  # custom validator lambda v: v.startswith('http')
https://www.apple.com/iphone.jpg

# Nice errors
>>> from pprint import pprint as pp
>>> pp(my_object.errors)
{
    17: {  # The row number
        'row': ['','',...]  # The complete row for reference,
        'errors': {  # Description of the errors
            'url': 'Validation failed',
            'currency': 'Invalid choice. Expected ['USD', 'YEN']. Got 'AUD' instead.
        }
    }
}
```

### Installation
    pip install smartcsv

### Usage

To see an entire set of usages check the `test` package (99% coverage).

The basic is to define a spec for the columns of your csv. Assuming the following CSV file:

title,category,subcategory,currency,price,url,image_url
    iPhone 5c blue,Phones,Smartphones,USD,399,http://apple.com/iphone,http://apple.com/iphone.jpg
    iPad mini,Tablets,Apple,USD,699,http://apple.com/iphone,http://apple.com/iphone.jpg

First you need to define the spec for your columns. This is an example (the one used in `tests`):

```python
CURRENCIES = ('USD', 'ARS', 'JPY')

COLUMNS_1 = [
    {'name': 'title', 'required': True},
    {'name': 'category', 'required': True},
    {'name': 'subcategory', 'required': False},
    {
        'name': 'currency',
        'required': True,
        'choices': CURRENCIES
    },
    {
        'name': 'price',
        'required': True,
        'validator': is_number
    },
    {
        'name': 'url',
        'required': True,
        'validator': lambda c: c.startswith('http')
    },
    {
        'name': 'image_url',
        'required': False,
        'validator': lambda c: c.startswith('http')
    },
]
```

You can then use `smartcsv` to parse the CSV:

```python
import smartcsv
with open('my-csv.csv', 'r') as f:
    reader = smartcsv.reader(f, columns=COLUMNS_1)
    for obj in reader:
        print(obj['title'])
```

`smartcsv.reader` uses the builtin `csv` module and accepts a dialect to use.

### More advanced usage

**Errors**

By default `smartcsv` will raise a `smartcsv.exceptions.InvalidCSVException` when it encounters an error in a column (a missing required field, a field different than choices, a validation failure, etc). The exception will have a nice error message in that case:

```python
# Assuming the price field is missing
try:
    item = next(reader)
except InvalidCSVException as e:
    print(e.errors)
    # {'price': 'Field required and not provided.'}
```

You can always avoid fast-failure (raising an exception on failure). You can pass the `fail_fast` argument as `False`. That will prevent exceptions, instead the errors are reported in the reader object (indicating the row number and the detail of the errors). For example, assuming a CSV with the an error in the second row:

```python
reader = smartcsv.reader(f, columns=COLUMNS_1, fail_fast=False)
for obj in reader:
    # All the processing is done Ok without exceptions raised.
    print(obj['title'])
    
error_row = reader.errors['rows'][1]  # Second row has index = 1. Errors are 0-indexed.
print(error_row['row'])  # Print original row data
print(error_row['errors'].keys())  # currency  (the currency column)
print(error_row['errors']['currency'])  # Invalid currency... (nice error explanation)
```

You can also specify a `max_failures` parameter. It will count failures and will raise an exception when that threshold is exceeded.

**Strip white spaces**

By default the `strip_white_spaces` option is set to True. Example:

```
sample.csv
title,price
   Some Product  ,  55.5  
```

`row['title']` will be "Some Product" and `row['price']` will be "55.5" (spaces stripped)

**Skip lines**
```
sample.csv
GENERATED BY AWESOME SCRIPT
2014-08-12

title,price
Some Product,55.5
```

The first 3 lines don't contain any valuable data so we'll skip them.

```python
reader = smartcsv.reader(f, columns=COLUMNS_1, fail_fast=False, skip_lines=3)
for obj in reader:
    print(obj['title'])
```

**Break (stop) on occurrance of first error**

By default, value of `fail_fast` is `True`. You can also mention it explicitly with `fail_fast=True`. This will cause halting execution of reader() function as soon as it faces an error in the csv file. This error can be data mismatch in between your data specification and found value in csv file. Data-validation failure also trigger `fail_fast`.

```python
reader = smartcsv.reader(f, columns=COLUMNS_1, fail_fast=True)
for obj in reader:
    print(obj['title'])
```

### Contributing

Fork, code, watch your tests pass, submit PR.
To test:

```bash
$ python setup.py test  # Run tests in your venv
$ tox  # Make sure it passes in all versions.
```

### Integration tests

There are "integration" tests included under `tests/integration`. They are not run by the default test runner. The idea of those tests is to have real examples of use cases for `smartcsv` documented.
You'll have to run them manually:

```bash
py.test tests/integration/lpnk/test_lpnk.py
```