word2vec-spam-filter - GitLite

KO | EN

gitlite — search

#react #ai #llm #go #android #docker #macos #awesome #cli #awesome-list #vue #pytorch #ios #mcp

word2vec-spam-filter

(★ 148)

Using word vectors to classify spam messages

File Explorer

Download Latest Version (.zip)

- app.py
- generate.py
- model.py
- requirements.txt
- Testing.ipynb
- - index.html
- - tsconfig.json
- package-lock.json
- package.json
- webpack.config.js
.gitignore
clip.gif
LICENSE
Makefile
README.md

# Installation Guide

1. Get the code

git clone https://github.com/doodyparizada/word2vec-spam-filter

Downloads the entire project code from GitHub to your computer.

cd word2vec-spam-filter

Moves into the project folder you just downloaded.

2. Node.js

Easy Recommended

Prerequisites

Git Needed to download the project code from GitHub.
Node.js Node.js must be installed to use npm. The LTS version is recommended.

npm install

Downloads and installs the libraries listed in package.json.

npm run dev

Starts the development/run server.

✅ After running the command, open the address shown in the terminal (usually something like http://localhost:3000) in your browser.

Pulled directly from this repo's README.

3. Python

Easy

Prerequisites

Git Needed to download the project code from GitHub.
Python 3 On Windows, be sure to check 'Add Python to PATH' during installation.

pip install -r requirements.txt

Installs the Python libraries listed in requirements.txt (or similar).

✅ If it runs without errors and prints output in the terminal, it worked.

Pulled directly from this repo's README.

4. Make

Medium

Prerequisites

Git Needed to download the project code from GitHub.
Make Usually pre-installed on Linux/macOS. On Windows, install separately (e.g. via MSYS2 or WSL).

make

Compiles the code based on the generated build configuration to produce an executable.

✅ If it finishes without errors, it worked. Try running the generated executable directly.

Pulled directly from this repo's README.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# word2vec-spam-filter

This is a project done during the [Kik](https://github.com/kikinteractive/) hackathon 2017.

In this project we demonstrate a way to classify spam messages on the client while protecting user privacy.

A client generates a "hash" from the message sending it to the server. The server then compares the "hash" to a bank of known reported messages.

The bank of known reported messages is created from spam reports. The server compares a given reported message to the previous bank of reported messages. If the message is similar to a previously reported message, a report count is incremented. Otherwise the message is added to the bank with a count of 1.

A message in the bank of reported messages is considered a spam message once it was reported more than 3 times.

## Preview

<img src="clip.gif" width="700"/>

## Corpus downloads
We used 2 datasets for creating sentence vectors:
1. word vectors taken from: https://github.com/stanfordnlp/GloVe
2. word frequencies from: https://github.com/IlyaSemenov/wikipedia-word-frequency/blob/master/results/enwiki-20150602-words-frequency.txt

## Configurable parameters (Hyper-Parameters)
We played around with a few configurations to get the best results for short user messages:

* Confidence Threshold - a number between 0.0 - 1.0 to determine when 2 messages are considered the same
* Distance Function - we used vector dot product
* Normalization - how to deal with words we don't have in our corpus, punctuation marks, non english words
* Vector Size - the longer the vector the higher the accuracy but heavier in memory
* Weight Function - given a word frequency, how to create the vector weights (`the` should weigh less than `camera`)
* Custom Corpus - creating the word vectors and frequencies from real user message data might yield better results
* Random Indices - how many random indices should the client send to the server to mask the original message indices

## Running the code
This project includes a single makefile to help with the initialization, dependency installation and corpus download.
You can invoke a help message by running:

```
make
```

Or you can manually run the server and client apps:

### server
In the `server` directory install the pip dependencies in a `virtualenv`:

```
pip install -r requirements.txt
```

and run the server:
```
python app.py
```

### web client
To use the web client go into the `webclient` directory in your terminal and then:
```
npm install
npm run dev
```

That should install all dependencies and kick start the project, if it all works you should see something like:
 > Project is running at http://localhost:3333/
 > webpack output is served from /

Now load http://localhost:3333/ in your browser

There are 3 different "view modes" which can be switched using the select box at the top right corner of the page.  
The 3 views are:
 * Standalone Tester: A textarea in which one can input a message and then either report it as spam or check whether it is classified as spam.
 * IM Sender: A textarea in which the user can input a message (or select a message from a bunch of existing ones) and then "send" the message to another client.
 * IM Receiver: A view which displays a list of received messages (using the `IM Sender`) and the ability to report each message as spam.