This repository has been archived on 2024-07-22. You can view files and clone it, but cannot push or open issues or pull requests.
Go to file
Fenpaws b6d0f4d63f rework_logging_#12 (#16)
Better and more extensive logging with proper logging levels, also new env variables are created to control those.

Reviewed-on: anthrove/e621-to-graph#16
Reviewed-by: Lennard Brinkhaus <lennard.brinkhaus@noreply.localhost>
Co-authored-by: Fenpaws <soxx@fenpa.ws>
Co-committed-by: Fenpaws <soxx@fenpa.ws>
2023-07-26 13:27:18 +00:00
.gitea/workflows refactor_folder_structure_#11 (#14) 2023-07-17 10:57:23 +00:00
build/package refactor_folder_structure_#11 (#14) 2023-07-17 10:57:23 +00:00
cmd/scraper rework_logging_#12 (#16) 2023-07-26 13:27:18 +00:00
deployments refactor_folder_structure_#11 (#14) 2023-07-17 10:57:23 +00:00
internal rework_logging_#12 (#16) 2023-07-26 13:27:18 +00:00
pkg rework_logging_#12 (#16) 2023-07-26 13:27:18 +00:00
.gitignore added dockerfile and compose config (#1) 2023-06-22 09:12:43 +00:00
go.mod refactor_folder_structure_#11 (#14) 2023-07-17 10:57:23 +00:00
go.sum go mod tidy 2023-06-21 13:32:22 +02:00
README.MD rework_logging_#12 (#16) 2023-07-26 13:27:18 +00:00
runMemgraphDev.cmd api-call-system-#10 (#13) 2023-07-17 08:10:13 +00:00

e621 to Graph

Scrapes users Favorite Posts and Upload them to a graph based database.

Prerequisites

Before running the program, ensure that you have the following installed on your system:

  • Go (version 1.20.2 or higher)
  • Docker (optional, if you plan to use Docker)

Installation

To install the project, follow these steps:

  1. Clone the repository:

    git clone https://git.dragse.it/fenpaws/e621-to-graph
    
  2. Change into the project's directory:

    cd e621-to-graph
    
  3. Install the required dependencies:

    go mod download
    go mod verify
    

Setting Up Environment Variables

The program requires certain environment variables to be set.

E621_API_KEY=
E621_USERNAME=
DB_TYPE=neo4j
DB_URL=
DB_PASSWORD=
DB_USERNAME=
# Allowed values are FATAL, ERROR, WARN, INFO, DEBUG, TRACE (default: INFO)
LOG_LEVEL=
# Allowed values are PLAIN, JSON (default: PLAIN)
LOG_FORMAT=
# Allowed values are TRUE, FALSE (default: FALSE)
NEO4J_DEBUG=

Running the Program

To run the program, execute the following command:

go run main.go

Building the Program

If you want to build the program into an executable binary, use the following command:

go build

Docker Support

The program can also be run using Docker. To use Docker, follow these steps:

  1. Create a .env file in the project's root directory with the following content:

    E621_API_KEY=
    E621_USERNAME=
    DB_TYPE=neo4j
    DB_URL=
    DB_PASSWORD=
    DB_USERNAME=
    # Allowed values are FATAL, ERROR, WARN, INFO, DEBUG, TRACE (default: INFO)
    LOG_LEVEL=
    # Allowed values are PLAIN, JSON (default: PLAIN)
    LOG_FORMAT=
    # Allowed values are TRUE, FALSE (default: FALSE)
    NEO4J_DEBUG=
    
  2. Build the Docker image:

    docker compose build
    
  3. Run the Docker container:

    docker compose up -d
    

Database Support

The program supports the following databases.

Neo4j

To improve performance, it is recommended to create indices on different nodes in Neo4j.

Run the following commands to create the required indices:

neo4j$ CREATE INDEX tagIndex FOR (t:e621Tag) ON (t.e621Tag);
neo4j$ CREATE INDEX postIndex FOR (p:e621Post) ON (p.e621PostID);
neo4j$ CREATE INDEX sourceUrlIndex FOR (s:Source) ON (s.URL);
neo4j$ CREATE INDEX userIdIndex FOR (u:e621User) ON (u.e621ID);

Memgraph:

To improve performance, it is recommended to create indices on different nodes in Memgraph.

Run the following commands to create the required indices:

memgraph> CREATE INDEX ON :e621Tag(e621Tag);
memgraph> CREATE INDEX ON :e621Post(e621PostID);
memgraph> CREATE INDEX ON :Source(URL);
memgraph> CREATE INDEX ON :e621User(e621ID);