TweetAnalysisWithSpark

KO | EN

gitlite — search

#javascript #react #ai #llm #go #rust #docker #awesome #awesome-list #security #pytorch #mcp #ios #openai

TweetAnalysisWithSpark

(★ 13)

Tweet Analysis with Spark

TweetAnalysisWithSpark 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Tweet Analysis with Spark

This tutorial shows how to use Spark to analyze crawls from Twitter's streaming API, which is formatted as a sequence of JSON objects, one per line. We assume CDH 5.3.0.

Fire up the Spark shell:

```
$ export HADOOP_CONF_DIR=/etc/hadoop/conf
$ spark-shell --master yarn-client --jars gson-2.3.1.jar --num-executors 10
```

The need for setting up the environment variable is due to [this issue](https://issues.cloudera.org/browse/DISTRO-664). You can download the jar [here](http://search.maven.org/#artifactdetails%7Ccom.google.code.gson%7Cgson%7C2.3.1%7Cjar).

If you want to play with individual tweets:

```
import com.google.gson._

val jsonParser = new JsonParser()
val gson = new GsonBuilder().setPrettyPrinting().create()

val raw = sc.textFile("/path/to/file");

// Take five JSON records and print each out.
for (r <- raw.take(5)) {
  println(gson.toJson(jsonParser.parse(r)))
}
```

Here's the [Gson API javadoc](http://google-gson.googlecode.com/svn/trunk/gson/docs/javadocs/index.html).

Now let's analyze the tweets at scale. Here's how we start:

```
import com.google.gson._

val raw = sc.textFile("/path/to/file");

var tweets =
  raw.map(line => {
    val jsonParser = new JsonParser()
    val obj = jsonParser.parse(line).getAsJsonObject()
    if (obj.has("delete")) {
      null
    } else {
      (obj.get("id").getAsLong(), obj.get("text").getAsString())
    }
  }).filter(x => x != null)
```

The variable `tweets` now holds an RDD of (tweetid, tweet text) pairs, with deletes filtered out.

Let's continue with word count:

```
val wc =
  tweets.flatMap(t => t._2.split(" ")).map(word => (word, 1)).reduceByKey(_ + _)
```

Let's say, find common terms, sort them, and save output to disk:

```
wc.filter(t => t._2 > 100).sortByKey().saveAsTextFile("wc_out")
```

TweetAnalysisWithSpark

TweetAnalysisWithSpark 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

TweetAnalysisWithSpark

TweetAnalysisWithSpark 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README