hadron

KO | EN

gitlite — search

#hacktoberfest #react #ai #typescript #android #machine-learning #rust #deep-learning #nodejs #cli #vue #security #openai #open-source

(★ 110)

Construct and run Hadoop MapReduce programs in Haskell

File Explorer

Download Latest Version (.zip)

- LineCount.hs
- WordCount.hs
- WordCountAuto.hs
- WordCountLocal.hs
- - Basic.hs
  - Conduit.hs
  - Controller.hs
  - Join.hs
  - Logger.hs
  - OutputFixer.hs
  - Protocol.hs
  - Run.hs
  - Streams.hs
  - Types.hs
  - Utils.hs
- Tests.hs
.ghci
.gitignore
.travis.yml
changelog.md
hadron.cabal
LICENSE
README.md
Setup.hs
stack-lts-2.yaml
stack-lts-3.yaml
stack.yaml
TODO.org

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Hadron - Hadoop MapReduce in Haskell [![Build Status](https://travis-ci.org/Soostone/hadron.svg?branch=master)](https://travis-ci.org/Soostone/hadron)

Hadron aims to bring Haskell's type-safety to the complex and delicate
world of Hadoop Streaming MapReduce.

## Features

* Ties into Hadoop via the Streaming interface

* Orchestrates multi-step Hadoop jobs so you don't have to manually
  call Hadoop at all.

* Provides typed interactions with your input/output data on hdfs, s3
  or any other system that Hadoop supports.

* Every Map-Reduce step is fully typed both on input and output,
  making a long, sophisticated sequence of jobs much easier to design
  and maintain.

* Built-in support for multi-way map-side joins. Disparate data
  sources are each mapped to a common, monoidal type which then gets
  `mconcat`ed during reduce by join key. We support both required
  (a-la inner) and optional (a-la outer) joins. Current shortcoming
  here is the loss of input typing; only Tap ByteString can be used on
  input in order to support multiple datasets.

* Various convenience combinators in the Controller module, covering
  common tasks.

## Shortcomings and Issues

Hadoop seems to be terrible at constantly changing little details,
program flags and behavior across major releases. While we try to make
this package as sound as possible, you may be forced to do some
debugging due to a difference in the way Hadoop works on the version
you are running.

This library has been most commonly tested on Amazon's EMR offering
and Cloudera's local demo VM.

## Status

hadron is used extensively by Soostone to process datasets with rows
in the billions. Improvement opportunities exist, but it is very much
functional.

## Modules

### Hadron.Basic

This module exposes low level functionality for constructing a single
MapReduce step. Not recommended for direct use in most cases.

### Hadron.Controller

High level module for automated orchestration of multi-stage MapReduce
jobs.

(More docs and examples to be added)

### Hadron.Protocol

Defines data encode/decode strategies via the Protocol type.

## TODO

See TODO.org.

# Contributors
* [Ozgun Ataman](http://github.com/ozataman)
* [Doug Beardsley](http://github.com/mightybyte)
* [Michael Xavier](http://github.com/MichaelXavier)
* [Thierry Bourrillon](https://github.com/thierry-b)

## Release Notes

## Version 0.5

- Hadoop operations are now put behind a Hadron.Run interface.

- Hadron.Run.Local now implements a basic form of Hadoop mimickery,
  allowing a wide variety of Controller-based MapReduce applications
  to be run locally on the development machine.

- Several interface improvements to working with local and HDFS based
  files during a Controller app.

- runOnce combinator allows running an "IO a" action in the central
  node and having the same value appear on the remote (worker) nodes.

hadron

File Explorer

Today's picks

Was this content helpful?

Recent Feedback

hadron

File Explorer

Today's picks

Was this content helpful?

Recent Feedback

Download README