119 lines
2.4 KiB
Markdown
119 lines
2.4 KiB
Markdown
# e621 to Graph
|
|
|
|
Scrapes users Favorite Posts and Upload them to a graph based database.
|
|
|
|
## Prerequisites
|
|
|
|
Before running the program, ensure that you have the following installed on your system:
|
|
|
|
- Go (version 1.20.2 or higher)
|
|
- Docker (optional, if you plan to use Docker)
|
|
|
|
## Installation
|
|
|
|
To install the project, follow these steps:
|
|
|
|
1. Clone the repository:
|
|
|
|
```shell
|
|
git clone https://git.dragse.it/fenpaws/e621-to-graph
|
|
```
|
|
|
|
2. Change into the project's directory:
|
|
|
|
```shell
|
|
cd e621-to-graph
|
|
```
|
|
|
|
3. Install the required dependencies:
|
|
|
|
```shell
|
|
go mod download
|
|
go mod verify
|
|
```
|
|
|
|
## Setting Up Environment Variables
|
|
|
|
The program requires certain environment variables to be set.
|
|
```plaintext
|
|
E621_API_KEY=
|
|
E621_USERNAME=
|
|
DB_TYPE=neo4j
|
|
DB_URL=
|
|
NEO4J_PASSWORD=
|
|
NEO4J_USERNAME=
|
|
```
|
|
|
|
## Running the Program
|
|
|
|
To run the program, execute the following command:
|
|
|
|
```shell
|
|
go run main.go
|
|
```
|
|
|
|
## Building the Program
|
|
|
|
If you want to build the program into an executable binary, use the following command:
|
|
|
|
```shell
|
|
go build
|
|
```
|
|
|
|
## Docker Support
|
|
|
|
The program can also be run using Docker. To use Docker, follow these steps:
|
|
|
|
1. Create a .env file in the project's root directory with the following content:
|
|
|
|
````plaintext
|
|
E621_API_KEY=
|
|
E621_USERNAME=
|
|
DB_TYPE=neo4j
|
|
DB_URL=
|
|
NEO4J_PASSWORD=
|
|
NEO4J_USERNAME=
|
|
````
|
|
|
|
2. Build the Docker image:
|
|
|
|
```shell
|
|
docker compose build
|
|
```
|
|
|
|
3. Run the Docker container:
|
|
|
|
```shell
|
|
docker compose up -d
|
|
```
|
|
|
|
## Database Support
|
|
|
|
The program supports the following databases.
|
|
|
|
### Neo4j
|
|
|
|
To improve performance, it is recommended to create indices on different nodes in Neo4j.
|
|
|
|
Run the following commands to create the required indices:
|
|
|
|
```shell
|
|
neo4j$ CREATE INDEX tagIndex FOR (t:e621Tag) ON (t.e621Tag);
|
|
neo4j$ CREATE INDEX postIndex FOR (p:e621Post) ON (p.e621PostID);
|
|
neo4j$ CREATE INDEX sourceUrlIndex FOR (s:Source) ON (s.URL);
|
|
neo4j$ CREATE INDEX userIdIndex FOR (u:e621User) ON (u.e621ID);
|
|
```
|
|
|
|
### Memgraph:
|
|
|
|
To improve performance, it is recommended to create indices on different nodes in Memgraph.
|
|
|
|
Run the following commands to create the required indices:
|
|
|
|
```shell
|
|
memgraph> CREATE INDEX ON :e621Tag(e621Tag);
|
|
memgraph> CREATE INDEX ON :e621Post(e621PostID);
|
|
memgraph> CREATE INDEX ON :Source(URL);
|
|
memgraph> CREATE INDEX ON :e621User(e621ID);
|
|
```
|