Archived

This repository has been archived on 2024-07-22. You can view files and clone it, but cannot push or open issues or pull requests.

Fenpaws b6d0f4d63f rework_logging_#12 (#16 )

Better and more extensive logging with proper logging levels, also new env variables are created to control those.

Reviewed-on: anthrove/e621-to-graph#16
Reviewed-by: Lennard Brinkhaus <lennard.brinkhaus@noreply.localhost>
Co-authored-by: Fenpaws <soxx@fenpa.ws>
Co-committed-by: Fenpaws <soxx@fenpa.ws>

2023-07-26 13:27:18 +00:00

2.9 KiB

Raw Blame History

e621 to Graph

Scrapes users Favorite Posts and Upload them to a graph based database.

Prerequisites

Before running the program, ensure that you have the following installed on your system:

Go (version 1.20.2 or higher)
Docker (optional, if you plan to use Docker)

Installation

To install the project, follow these steps:

Clone the repository:

git clone https://git.dragse.it/fenpaws/e621-to-graph

Change into the project's directory:
```
cd e621-to-graph
```
Install the required dependencies:
```
go mod download
go mod verify
```

Setting Up Environment Variables

The program requires certain environment variables to be set.

E621_API_KEY=
E621_USERNAME=
DB_TYPE=neo4j
DB_URL=
DB_PASSWORD=
DB_USERNAME=
# Allowed values are FATAL, ERROR, WARN, INFO, DEBUG, TRACE (default: INFO)
LOG_LEVEL=
# Allowed values are PLAIN, JSON (default: PLAIN)
LOG_FORMAT=
# Allowed values are TRUE, FALSE (default: FALSE)
NEO4J_DEBUG=

Running the Program

To run the program, execute the following command:

go run main.go

Building the Program

If you want to build the program into an executable binary, use the following command:

go build

Docker Support

The program can also be run using Docker. To use Docker, follow these steps:

Create a .env file in the project's root directory with the following content:

E621_API_KEY=
E621_USERNAME=
DB_TYPE=neo4j
DB_URL=
DB_PASSWORD=
DB_USERNAME=
# Allowed values are FATAL, ERROR, WARN, INFO, DEBUG, TRACE (default: INFO)
LOG_LEVEL=
# Allowed values are PLAIN, JSON (default: PLAIN)
LOG_FORMAT=
# Allowed values are TRUE, FALSE (default: FALSE)
NEO4J_DEBUG=

Build the Docker image:
```
docker compose build
```
Run the Docker container:
```
docker compose up -d
```

Database Support

The program supports the following databases.

Neo4j

To improve performance, it is recommended to create indices on different nodes in Neo4j.

Run the following commands to create the required indices:

neo4j$ CREATE INDEX tagIndex FOR (t:e621Tag) ON (t.e621Tag);
neo4j$ CREATE INDEX postIndex FOR (p:e621Post) ON (p.e621PostID);
neo4j$ CREATE INDEX sourceUrlIndex FOR (s:Source) ON (s.URL);
neo4j$ CREATE INDEX userIdIndex FOR (u:e621User) ON (u.e621ID);

Memgraph:

To improve performance, it is recommended to create indices on different nodes in Memgraph.