yacy_grid_loader - GitLite

KO | EN

gitlite — search

#python #react #ai #typescript #llm #go #golang #docker #macos #nodejs #awesome #cli #pytorch #openai

yacy_grid_loader

(★ 613)

A headless browser as loader microservice in the YaCy Grid

File Explorer

Download Latest Version (.zip)

- FUNDING.yml
- org.eclipse.buildship.core.prefs
- org.eclipse.jdt.core.prefs
- restart.sh
- start.sh
- start_loader_docker.sh
- stop.sh
- config.properties
- - gradle-wrapper.jar
  - gradle-wrapper.properties
- yacy_grid_mcp
.classpath
.dockerignore
.gitignore
.gitmodules
.project
build.gradle
Dockerfile
gradlew
gradlew.bat
README.md

# Installation Guide

1. Get the code

git clone https://github.com/yacy/yacy_grid_loader

Downloads the entire project code from GitHub to your computer.

cd yacy_grid_loader

Moves into the project folder you just downloaded.

2. Docker

Easy Recommended

Prerequisites

Git Needed to download the project code from GitHub.
Docker Desktop Needed to build and run containers. Install it and keep it running in the background.

docker build -t yacy-grid-loader .

Builds a runnable image based on the Dockerfile.

docker run -p 8080:80 yacy-grid-loader

Runs the built image as an actual container.

✅ Run docker compose ps to check the containers are Up. If the README mentions a port, open http://localhost:PORT in your browser.

3. Gradle (Java/Kotlin)

Medium

Prerequisites

Git Needed to download the project code from GitHub.
JDK (Java) Required to build and run Java/Kotlin projects.
Gradle If the repo includes gradlew (Gradle Wrapper), you don't need to install Gradle separately.

> gradle run

Builds the project using Gradle.

✅ A BUILD SUCCESSFUL message means it worked. The output is created under build/.

Pulled directly from this repo's README.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# YaCy Grid Component: Loader

The YaCy Grid is the second-generation implementation of YaCy, a peer-to-peer search engine.
A YaCy Grid installation consists of a set of micro-services which communicate with each other
using the MCP, see https://github.com/yacy/yacy_grid_mcp

## Purpose

The Loader is a microservice which can be deployed i.e. using Docker.
Each search engine needs a file loader and this component will do that work.
The special feature of this loader is it's embedded headless browser which makes
it possible to load rich content and provide that content for a search engine.

## What it does

When the Loader component is started, it searches for a MCP and connects to it.
By default the local host is searched for a MCP but you can configure one yourself.

The Loader will then wait for client requests and performs web loading upon request.
It also has a MCP queue listener to react on loading requests in the working queues.
After loading of content the loader will push back results to the MCP storage and puts
another message on the MCP message queue to process the loaded content.

## Installation: Download, Build, Run
At this time, yacy_grid_parser is not provided in compiled form, you easily build it yourself. It's not difficult and done in one minute! The source code is hosted at https://github.com/yacy/yacy_grid_loader, you can download and run it with:

> git clone --recursive https://github.com/yacy/yacy_grid_loader.git

If you just want to make a update, do the following

> git pull origin master
    > git submodule foreach git pull origin master

To build and start the loader, run

> cd yacy_grid_loader
    > gradle run

Please read also https://github.com/yacy/yacy_grid_mcp/blob/master/README.md for further details.

## Contribute

This is a community project and your contribution is welcome!

1. Check for [open issues](https://github.com/yacy/yacy_grid_loader/issues)
   or open a fresh one to start a discussion around a feature idea or a bug.
2. Fork [the repository](https://github.com/yacy/yacy_grid_loader.git)
   on GitHub to start making your changes (branch off of the master branch).
3. Write a test that shows the bug was fixed or the feature works as expected.
4. Send a pull request and bug us on Gitter until it gets merged and published. :)

## What is the software license?
LGPL 2.1

Have fun!

@0rb1t3r