This repository has been archived on 2024-07-22. You can view files and clone it, but cannot push or open issues or pull requests.
Go to file
SoXX 3be16a9277 new_scrape_algorithm_#5 (#20)
First implementation of the new algorithm that got proposed in issue #5

Reviewed-on: anthrove/e621-to-graph#20
Reviewed-by: Lennard Brinkhaus <lennard.brinkhaus@noreply.localhost>
Reviewed-by: daskadse <daskadse@noreply.localhost>
Co-authored-by: SoXX <soxx@fenpa.ws>
Co-committed-by: SoXX <soxx@fenpa.ws>
2023-11-15 20:28:44 +00:00
.gitea/workflows implement_new_e621_sdk_#17 (#18) 2023-11-08 13:01:27 +00:00
build/package implement_new_e621_sdk_#17 (#18) 2023-11-08 13:01:27 +00:00
cmd/scraper implement_new_e621_sdk_#17 (#18) 2023-11-08 13:01:27 +00:00
deployments implement_new_e621_sdk_#17 (#18) 2023-11-08 13:01:27 +00:00
internal new_scrape_algorithm_#5 (#20) 2023-11-15 20:28:44 +00:00
pkg new_scrape_algorithm_#5 (#20) 2023-11-15 20:28:44 +00:00
.gitignore added dockerfile and compose config (#1) 2023-06-22 09:12:43 +00:00
go.mod implement_new_e621_sdk_#17 (#18) 2023-11-08 13:01:27 +00:00
go.sum implement_new_e621_sdk_#17 (#18) 2023-11-08 13:01:27 +00:00
README.MD rework_logging_#12 (#16) 2023-07-26 13:27:18 +00:00
runMemgraphDev.cmd api-call-system-#10 (#13) 2023-07-17 08:10:13 +00:00

e621 to Graph

Scrapes users Favorite Posts and Upload them to a graph based database.

Prerequisites

Before running the program, ensure that you have the following installed on your system:

  • Go (version 1.20.2 or higher)
  • Docker (optional, if you plan to use Docker)

Installation

To install the project, follow these steps:

  1. Clone the repository:

    git clone https://git.dragse.it/fenpaws/e621-to-graph
    
  2. Change into the project's directory:

    cd e621-to-graph
    
  3. Install the required dependencies:

    go mod download
    go mod verify
    

Setting Up Environment Variables

The program requires certain environment variables to be set.

E621_API_KEY=
E621_USERNAME=
DB_TYPE=neo4j
DB_URL=
DB_PASSWORD=
DB_USERNAME=
# Allowed values are FATAL, ERROR, WARN, INFO, DEBUG, TRACE (default: INFO)
LOG_LEVEL=
# Allowed values are PLAIN, JSON (default: PLAIN)
LOG_FORMAT=
# Allowed values are TRUE, FALSE (default: FALSE)
NEO4J_DEBUG=

Running the Program

To run the program, execute the following command:

go run main.go

Building the Program

If you want to build the program into an executable binary, use the following command:

go build

Docker Support

The program can also be run using Docker. To use Docker, follow these steps:

  1. Create a .env file in the project's root directory with the following content:

    E621_API_KEY=
    E621_USERNAME=
    DB_TYPE=neo4j
    DB_URL=
    DB_PASSWORD=
    DB_USERNAME=
    # Allowed values are FATAL, ERROR, WARN, INFO, DEBUG, TRACE (default: INFO)
    LOG_LEVEL=
    # Allowed values are PLAIN, JSON (default: PLAIN)
    LOG_FORMAT=
    # Allowed values are TRUE, FALSE (default: FALSE)
    NEO4J_DEBUG=
    
  2. Build the Docker image:

    docker compose build
    
  3. Run the Docker container:

    docker compose up -d
    

Database Support

The program supports the following databases.

Neo4j

To improve performance, it is recommended to create indices on different nodes in Neo4j.

Run the following commands to create the required indices:

neo4j$ CREATE INDEX tagIndex FOR (t:e621Tag) ON (t.e621Tag);
neo4j$ CREATE INDEX postIndex FOR (p:e621Post) ON (p.e621PostID);
neo4j$ CREATE INDEX sourceUrlIndex FOR (s:Source) ON (s.URL);
neo4j$ CREATE INDEX userIdIndex FOR (u:e621User) ON (u.e621ID);

Memgraph:

To improve performance, it is recommended to create indices on different nodes in Memgraph.

Run the following commands to create the required indices:

memgraph> CREATE INDEX ON :e621Tag(e621Tag);
memgraph> CREATE INDEX ON :e621Post(e621PostID);
memgraph> CREATE INDEX ON :Source(URL);
memgraph> CREATE INDEX ON :e621User(e621ID);