go-nlp - GitLite

KO | EN

gitlite — search

#javascript #python #typescript #android #rust #docker #nodejs #java #windows #vue #awesome-list #pytorch #ios #openai

go-nlp

(★ 95)

Utilities for working with discrete probability distributions and other tools useful for doing NLP work

File Explorer

Download Latest Version (.zip)

- counter.go
- counter_test.go
- Makefile
- - main.go
  - Makefile
  - maxent.go
- - .gitignore
  - main.go
  - Makefile
  - naivebayes.go
- Makefile
- words.go
- blas.go
- countervector.go
- frozencounter.go
- keyset.go
- Makefile
- common.go
- Makefile
- gradient_descent.go
- Makefile
- Makefile
- README.md
- smoothing.go
.gitignore
build.sh
README.md
test.sh

# Installation Guide

1. Get the code

git clone https://github.com/nuance/go-nlp

Downloads the entire project code from GitHub to your computer.

cd go-nlp

Moves into the project folder you just downloaded.

2. Make

Medium Recommended

Prerequisites

Git Needed to download the project code from GitHub.
Make Usually pre-installed on Linux/macOS. On Windows, install separately (e.g. via MSYS2 or WSL).

cd counter

This project's files live in a subfolder, so move into it first.

make

Compiles the code based on the generated build configuration to produce an executable.

✅ If it finishes without errors, it worked. Try running the generated executable directly.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

GNLP
====

A few structures for doing NLP analysis / experiments.

Basics
------

* counter.Counter

A map-like data structure for representing discrete probability
distributions. Contains an underlying map of event -> probability
along with a probability for all other events. Supports some
element-wise mathematical operations with other counter.Counter
objects.

```go
// Create a counter with 0 probability for unknown events (and with ""
// corresponding to the unknown event)
balls := counter.New(0.0)
	
// Add some observations
balls.Incr("blue")
balls.Incr("blue")
balls.Incr("red")

// Normalize into a discrete distribution
balls.Normalize()

// blue => 0.666666
balls.Get("blue")

// purple => 0.0
balls.Get("purple")

preference = counter.New(0.0)
preference.Set("red", 2.0)
preference.Set("blue", 1.0)
preference.Normalize()

expected_with_preference = counter.Multiply(balls, preference)
expected_with_preference.Normalize()

// blue => 0.5
expected_with_preference.Get("blue")
// red => 0.5
expected_with_preference.Get("red")

// You can also use log probabilities
balls.LogNormalize()
preferences.LogNormalize()

// And do in-place operations
balls.Add(preferences)

// Log-normalize expects counters with positive counts, so
// exponentiate-then-normalize
balls.Exp()
balls.LogNormalize()

// blue => -1 (== lg(0.5))
balls.Get("blue")
```

* frozencounter.Counter

Similar to counter.Counters, but with a fixed set of keys and no
default value. Represented under the hood as an array of doubles (with
order fixed according to the set of keys). Supports element-wise math
operations with other frozencounter.Counters that share the same set
of keys. Some mathematical operations are accelerated by the BLAS
library.

```go
fBalls := frozencounter.Freeze(balls)
fPrefs := frozencounter.Freeze(preference)

fExpectedWithPreference := frozencounter.Multiply(fBalls, fPrefs)
```