subreddit-analytics
End to end data pipeline to extract and analyze submissions from any subreddit using Pushshift, python, dbt and BigQuery.
File Explorer
- extensions.json
- settings.json
- post_ids.csv
- .gitkeep
- .gitkeep
- remove_stopwords.sql
- tfidf.sql
- __subreddit_data_raw__sources.yml
- _base_subreddit_data_raw__comments.yml
- _base_subreddit_data_raw__submissions.yml
- base_subreddit_data_raw__comments.sql
- base_subreddit_data_raw__submissions.sql
- _dim_words.yml
- _int_submissions_score_adjusted.yml
- _tfidf_submissions_and_comments.yml
- dim_words.sql
- int_submissions_score_adjusted.sql
- tfidf_submissions_and_comments.sql
- .gitkeep
- .gitkeep
- dbt_utils.py
- .gitkeep
- .gitignore
- dbt_project.yml
- packages.yml
- profiles.yml
- README.md
- Taskfile.yml
- dashboard.gif
- itcrowd.gif
- vocabulary.png
- main.tf
- variables.tf
- blocks.py
- flows_api_to_bq.py
- tasks_get_subreddit_data.py
- tasks_upload_to_bq.py
- utils_.py
- .env.example
- .flake8
- .gitignore
- .sqlfluff
- .sqlfluffignore
- LICENSE
- poetry.lock
- poetry.toml
- pyproject.toml
- README.md
- Taskfile.yml
# Use via CDN
jsDelivrjsDelivr serves any public GitHub repository as a CDN with zero setup. Pick a version and a file to get a ready-to-paste link and snippet.
Command Glossary
Commands referenced in this DOCs, explained below.
brew install
View Details ▼
brew install
Install a Homebrew formula or cask.
brew install {{formula|cask}}
Install a formula/cask:
brew install {{[-s|--build-from-source]}} {{formula}}
Build and install a formula from source (dependencies will still be installed from bottles):
brew install {{[-n|--dry-run]}} {{formula|cask}}
Download the manifest, print what would be installed but don't actually install anything:
dbt
View Details ▼
dbt
A tool to model transformations in data warehouses.
dbt debug
Debug the dbt project and the connection to the database:
dbt run
Run all models of the project:
dbt test --select example_model
Run all tests of `example_model`:
gcloud auth
View Details ▼
gcloud auth
Grant and revoke authorization to `gcloud` and manage credentials.
See also: `gcloud`.
gcloud auth login
Authorize Google Cloud access for the `gcloud` CLI with Google Cloud user credentials and set the current account as active:
gcloud auth activate-service-account
Authorize Google Cloud access similar to `gcloud auth login` but with service account credentials:
gcloud auth application-default
Manage Application Default Credentials (ADC) for Cloud Client Libraries:
gcloud projects
View Details ▼
gcloud projects
Manage project access policies in Google Cloud.
See also: `gcloud`.
gcloud projects create {{project_id|project_number}}
Create a new project:
gcloud projects list
List all active projects:
gcloud projects describe {{project_id}}
Display metadata for a project:
poetry install
View Details ▼
poetry install
Install all dependencies for a Python project as defined in the pyproject.toml file.
poetry install
Install dependencies:
poetry install --no-root
Skip installing the project itself as a dependency:
poetry install --without dev
Install only production dependencies:
poetry shell
View Details ▼
poetry shell
Spawn a shell within the virtual environment.
Note: The `shell` command was moved to a plugin: `poetry-plugin-shell`.
poetry shell
Spawn a shell within the virtual environment:
python
View Details ▼
python
Python language interpreter.
python
Start a REPL (interactive shell):
python {{path/to/file.py}}
Execute a specific Python file:
python -i {{path/to/file.py}}
Execute a specific Python file and start a REPL:
terraform apply
View Details ▼
terraform apply
Create or update infrastructure according to Terraform configuration files.
terraform apply
Create or update infrastructure:
terraform apply -auto-approve
Create or update infrastructure, skipping interactive approval:
terraform apply {{path/to/file.tfplan}}
Apply a plan file:
terraform init
View Details ▼
terraform init
Initialize a new or existing Terraform working directory.
terraform init
Initialize the current working directory:
terraform init -upgrade
Initialize and upgrade modules and providers to the latest allowed versions:
terraform init -reconfigure
Initialize and reconfigure the backend, ignoring any saved configuration:
terraform plan
View Details ▼
terraform plan
Generate and show Terraform execution plans.
terraform plan
Generate and show the execution plan in the current directory:
terraform plan -destroy
Show a plan to destroy all remote objects that currently exist:
terraform plan -refresh-only
Show a plan to update the Terraform state and output values:
