paws - GitLite

KO | EN

gitlite — search

#python #typescript #llm #go #android #machine-learning #docker #macos #windows #cli #vue #security #mcp #openai

paws

(★ 54)

PAWS: A Web Shell

파일 탐색기

최종 버전 다운로드 (.zip)

- - jobber.yaml
  - linter.yaml
  - maintain-dumps-nfs.yaml
  - minesweeper.yaml
  - nbserve.yaml
  - notification.yaml
  - paws-hub.yaml
  - renderer.yaml
  - singleuser.yaml
  - tofu-fmt.yaml
  - update-container-tags.yaml
- - csi-secret-cinderplugin.yaml.codfw1dev.crypt
  - csi-secret-cinderplugin.yaml.eqiad1.crypt
  - sc.yaml
- - prometheus-ingress.yaml.j2
- - codfw1dev.yaml
  - eqiad1.yaml
- ansible.cfg
- paws.yaml
- - Dockerfile
  - README.md
- - Dockerfile
- - Dockerfile
  - LICENSE
  - README.md
  - requirements.in
  - requirements.txt
- - Dockerfile
  - README.md
  - robots.txt
- - Dockerfile
  - paws-favicon.ico
  - PAWS.svg
  - README.md
- - basic.tpl
  - Dockerfile
  - full.tpl
  - README.md
  - renderer.py
  - requirements.txt
- - banner
  - Dockerfile
  - hide_clusters_tab.css
  - install-extensions
  - install-julia
  - install-sql-tool
  - requirements.txt
  - user-config.py
  - user-fixes.py
- - frontpage.yaml
  - localdev.yaml
  - NOTES.txt
  - public.yaml
- .helmignore
- Chart.yaml
- codfw-secrets.yaml
- codfw.yaml
- production.yaml
- secrets.yaml
- values.yaml
- - Dockerfile
- - Dockerfile
- 127d.tf
- 135a.tf-magnum-capi
- codfw1dev-backend.conf
- eqiad1-backend.conf
- main.tf
- secrets.tf
- vars.tf
- yamllint.sh
.flake8
.gitattributes
.gitignore
.gitmodules
.yamllint.conf
build.py
deploy.sh
LICENSE
pyproject.toml
README.md
secrets-codfw1dev.sh
secrets-eqiad1.sh
toolinfo.json
tox.ini

# 설치 가이드

1. 코드 내려받기

git clone https://github.com/toolforge/paws

깃허브에서 프로젝트 코드 전체를 내 컴퓨터로 내려받습니다.

cd paws

방금 내려받은 프로젝트 폴더 안으로 이동합니다.

2. Docker

쉬움 추천

사전 준비물

Git GitHub에서 프로젝트 코드를 내려받으려면 필요합니다.
Docker Desktop 컨테이너를 빌드하고 실행하려면 필요합니다. 설치 후 실행해서 백그라운드에 켜두세요.

`minikube start --driver=docker --kubernetes-version=v1.20.11`

이 명령어를 터미널에 그대로 입력해 실행하세요.

✅ 터미널에 docker compose ps 를 입력해 컨테이너들이 Up 상태인지 확인하세요. README에 포트 번호가 적혀있다면 브라우저에서 http://localhost:포트번호 로 접속해보세요.

이 레포의 README에 적힌 실제 명령어를 그대로 가져왔습니다.

3. Python

쉬움

사전 준비물

Git GitHub에서 프로젝트 코드를 내려받으려면 필요합니다.
Python 3 설치 시 'Add Python to PATH' 옵션을 꼭 체크하세요 (Windows).

pip install -r images/minesweeper/requirements.txt

requirements.txt 등에 명시된 파이썬 라이브러리를 설치합니다.

python <실행할 파일명>.py # README에서 정확한 실행 파일명을 확인하세요

파이썬 스크립트(또는 모듈)를 실행합니다.

✅ 에러 메시지 없이 실행되고 터미널에 안내 문구가 출력되면 정상입니다.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# PAWS

PAWS: A Web Shell (PAWS) is a Jupyter notebooks deployment that has been customized to make interacting with Wikimedia wikis easier. It allows users to create and share documents that contain live code, visualizations such as graphs, rich text, etc. The user created notebooks are a powerful tool that enables data analysis and scientific research, and also transforms the way in which programmers write code - by enabling an exploratory environment with a quick feedback loop, and a low barrier for entry through it's easy to use graphical interface.

## Contributing

Bugs, issues and feature requests are found on [Wikimedia Foundation's Phabricator](https://phabricator.wikimedia.org/).
There is a [workboard](https://phabricator.wikimedia.org/project/view/1648/) and a project tag of `#paws` to use for related work. You can reference code and commits from this repo at the Phabricator mirror of the code [here](https://phabricator.wikimedia.org/diffusion/PAWS/browse/main/), but please do not clone or try to use that mirror directly.

To contribute to this project's code, please fork the repo on [GitHub](https://github.com/toolforge/paws/) and submit a pull request.

If you have push access to the project, we ask that new changes be reviewed by one other
project member by using either a feature branch on the https://github.com/toolforge/paws repo
to trigger a pull request or using a fork to set up a pull request.

### Pull Requests and CI

When a pull request is opened a few things are run automatically. Any container that was modified in /images will be built and pushed to quay.io. Your branch will be updated with an additional commit, updating the values.yaml file to point to the new image tags. And a linter will be run. These workflows, and their status, will be visible in the github pull request page. At this point you, or anyone else, will be able to pull down the branch in the PR and run it locally in minikube as described below.

If your PR originates from a fork, please be sure "Allow edits and access to secrets by maintainers" is enabled such that the CI can function. Alternatively please manually edit the values.yaml to match the PR number for any containers that your code updates.

### Settings up a development environment

It is possible to run a fully-functioning PAWS system inside [minikube](https://minikube.sigs.k8s.io/docs/)! You don't need
access to the secrets.yaml file to do it either, since the defaults mostly support it.

You will need to install minikube (tested on minikube v1.33.1) and [helm](https://helm.sh) and kubectl on your system. When you are confident those are working, start minikube with:
 - `minikube start --kubernetes-version=v1.27.8`
 - `minikube addons enable ingress`
(from the top level of this repo):
install the dependencies for the PAWS dev environment with these steps:
 - `helm repo add jupyterhub https://jupyterhub.github.io/helm-chart/`
 - `helm dep up paws/`
 - `kubectl create namespace paws-dev`
 - `helm -n paws-dev upgrade --install dev paws/ --timeout=50m`

The rest of the setup instructions will display on screen as long as the install is successful.
Please refer to the helm documentation from there.

If you are experiencing issues with the installation, you can try changing the driver configuration in minikube: https://minikube.sigs.k8s.io/docs/drivers/

- First delete the current cluster:

`minikube delete`

- Start a new cluster with the driver you want to use (e.g. docker, virtualbox, hyperkit, etc.):

`minikube start --driver=docker --kubernetes-version=v1.20.11`

Another possible solution if minikube is acting weird might be to upgrade minikube, or even to
increase the default memory:

`minikube config set memory 4096`

#### Working with images
Keep in mind that opening a PR will, attempt to, build any image that has changed in the PR branch. This method is fine to build and test the resulting container. Though it is often easier to build, and rebuild, a container locally for testing. The following describes how to build and use a container locally.

Definitions for the PAWS images are found in the images/ directory. If you start a dev environment, it will pull those images from quay.io by default, just like in Wikimedia Cloud Services. If you are making changes to the images and testing those locally, you'll need to build them and tag them for your local environment, possibly setting them in your local values file with the tags you set.

If you are using minikube, you need to make sure you are using minikube's docker, not your system's docker with `eval $(minikube docker-env)`. Now your docker commands will operate on the minikube environment.

For example, let's say you wanted to update the singleuser image (which is the actual notebook server image):
- `cd images/singleuser`
- `docker build -t tag-you-are-going-to-use:whatever .`

And then you should have the image with a tag of `tag-you-are-going-to-use:whatever` that you could edit into your values.yaml file for local helm work.
## Useful libraries
### Accessing Database Replicas With Pandas and Sqlalchemy

Pandas is a lovely high level library for in-memory data manipulations. In order to get the result of a SQL query as a pandas dataframe use:
```
from sqlalchemy import create_engine
import sys, os
import pandas as pd

constr = 'mysql+pymysql://{user}:{pwd}@{host}'.format(user=os.environ['MYSQL_USERNAME'],
                                                      pwd=os.environ['MYSQL_PASSWORD'],
                                                      host=os.environ['MYSQL_HOST'])
con = create_engine(constr)

df = pd.read_sql('select * from plwiki_p.logging limit 10', con)
```

### Storage space
Publishing space

A notebook can be turned into a public notebook by publishing a link to it. This works as the notebook is made available in a read only mode. An example might be …revisions-sql.ipynb?kernel_name=python3. It could be wise to add the kernel name to the link, even if it isn't necessary in some cases.

If you want to run the copy yourself, or do interactive changes, you must download the notebook and reupload on your own account. Downloading the raw format of the previous example can be done by adding format=raw to the previous example …revisions-sql.ipynb?format=raw. This download-reupload-process is somewhat awkward.

Note that a notebook will always be published, as the link can be guessed, so don't add any private information.

### Running tests locally
PAWS tests are managed through docker. You can find all of the tests in the tests directory. If you want to build and run them locally run:
`docker build -f tests/<testname>/Dockerfile -t test:01 . ; docker run test:01`
Where <testname> is any of the directories in tests. For example:
`docker build -f tests/tox/Dockerfile -t test:01 . ; docker run test:01`
Would run tox.

### To know more about PAWS have a look at:
https://wikitech.wikimedia.org/wiki/PAWS

### Comment to Phabricator
To have a PR make comments to an associated phabricator ticket have the last line of the commit look like:

Bug: <ticket number>

For example:
Bug: T318182

### Deployment ###
```
bash deploy.sh <eqiad1|codfw1dev>
```

update the web proxy in horizon to point to current cluster.

https://wikitech.wikimedia.org/wiki/PAWS/Admin#Deployment

#### Disaster Recovery
If the entire project is removed two parts of paws are not managed by tofu/ansible.
Object storage container: An object storage container named "tofu-state" will need to be generated in horizon. This is where the state file for tofu resides.
NFS: The NFS server is not included. And a fresh NFS server will be needed for paws to operate.

# backup prometheus
see ansible/files/prometheus-data.sh for example of backup/restore