WildNLP - GitLite

KO | EN

gitlite — search

#codex #cli #chatgpt #linux #mcp #monitoring #electron #git #large-language-models #pdf

WildNLP

(★ 76)

No description available.

File Explorer

ZIP Download

- - aspects.rst
  - conf.py
  - datasets.rst
  - developers.rst
  - index.rst
  - quickstart.rst
- Makefile
- - qwerty.csv
  - qwerty.png
  - remove_char.csv
  - remove_char.png
- evaluation.py
- README.md
- requirements.txt
- _cl_config.yaml
- index.html
- out.json
- - test_all.py
  - test_article.py
  - test_change_char.py
  - test_compose.py
  - test_digits2words.py
  - test_misspellings.py
  - test_punctuation.py
  - test_qwerty.py
  - test_remove_char.py
  - test_reverser.py
  - test_sentiment_masking.py
  - test_sub_string.py
  - test_swap.py
  - test_white_spaces.py
- - test_conll.py
  - test_imdb.py
  - test_sample_dataset.py
  - test_snli.py
  - test_squad.py
- - __init__.py
  - add_sub_string.py
  - articles.py
  - base.py
  - change_char.py
  - digits2words.py
  - misspelling.py
  - punctuation.py
  - qwerty.py
  - remove_char.py
  - sentiment_masking.py
  - swap.py
  - utils.py
  - white_spaces.py
- - __init__.py
  - base.py
  - conll.py
  - imdb.py
  - sample_dataset.py
  - snli.py
  - squad.py
- __init__.py
.gitignore
.travis.yml
Dockerfile
logo.png
Makefile
MANIFEST.in
README.md
requirements.txt
setup.py

# Use via CDN

jsDelivr

jsDelivr serves any public GitHub repository as a CDN with zero setup. Pick a version and a file to get a ready-to-paste link and snippet.

Version

File

Link

Example

Command Glossary

Commands referenced in this DOCs, explained below.

🔍

`pip install`

View Details ▼

Install Python packages.

pip install {{package1 package2 ...}}

Install one or more packages:

pip install {{package1 package2 ...}} {{[-U|--upgrade]}}

Upgrade all specified packages to the latest version, installing any that are not already present:

pip install {{package}}=={{version}}

Install a specific version of a package:

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

[![Documentation Status](https://readthedocs.org/projects/nlp-demo/badge/?version=latest)](https://nlp-demo.readthedocs.io/en/latest/?badge=latest)
[![Downloads](https://pepy.tech/badge/wild-nlp)](https://pepy.tech/project/wild-nlp)
[![PyPI version](https://badge.fury.io/py/wild-nlp.svg)](https://badge.fury.io/py/wild-nlp)

![alt wildnlp-logo](https://raw.githubusercontent.com/MI2DataLab/WildNLP/master/logo.png)

Corrupt an input text to test NLP models' robustness.  
For details refer to https://nlp-demo.readthedocs.io

## Installation
`pip install wild-nlp`

## Supported aspects
All together we defined and implemented 11 aspects of text corruption.

1. **Articles**
   
   Randomly removes or swaps articles into wrong ones.

2. **Digits2Words**

Converts numbers into words. Handles floating numbers as well.

3. **Misspellings**

Misspells words appearing in the Wikipedia list of:  
    * **commonly misspelled English words**  
    * **homophones**

4. **Punctuation**

Randomly adds or removes specified punctuation marks.

5. **QWERTY**

Simulates errors made while writing on a QWERTY-type keyboard.

6. **RemoveChar**

Randomly removes:  
   * **characters** from words or  
   * **white spaces** from sentences

7. **SentimentMasking**

Replaces random, single character with for example an asterisk in:  
   * **negative** or  
   * **positive** words from Opinion Lexicon:    
   http://www.cs.uic.edu/~liub/FBS/sentiment-analysis.html

8. **Swap**

Randomly swaps two characters within a word, excluding punctuations.
   
9. **Change char**

Randomly change characters according to chosen dictionary, default is 'ocr' to simulate simple OCR errors.
   
10. **White spaces**

Randomly add or remove white spaces (listed as a parameter).

11. **Sub string**

Randomly add a substring to simulate more comples signs.
   
   
```diff
- All aspects can be chained together with the wildnlp.aspects.utils.compose function.
```

## Supported datasets
Aspects can be applied to any text. Below is the list of datasets for which we already implemented processing pipelines.

1. **CoNLL**

The CoNLL-2003 shared task data for language-independent named entity recognition.

2. **IMDB**

The IMDB dataset containing movie reviews for a sentiment analysis. The dataset consists of 50 000 reviews of two classes, negative and positive.

3. **SNLI**

The SNLI dataset supporting the task of natural language inference.

4. **SQuAD**

The SQuAD dataset for the Machine Comprehension problem.

## Usage
```python
from wildnlp.aspects.dummy import Reverser, PigLatin
from wildnlp.aspects.utils import compose
from wildnlp.datasets import SampleDataset

dataset = SampleDataset()
dataset.load()

# Crate a composed corruptor function.
# Functions will be applied in the same order they appear.
composed = compose(Reverser(), PigLatin())

# Apply the function to the dataset
modified = dataset.apply(composed)
```

# Acknowledgments

Adam Slucki and Dominika Basaj were financially supported by grant POIR.01.01.01-00-0328/17-01.
Przemyslaw Biecek was financially supported by NCN Opus grant 2017/27/B/ST6/0130.