goblin-store/README.md

Goblin Store

Incremental backup tool using rsync and rclone.

Usage

Goblin store provides two options to backup a directory - cloud and local.

Local Backups

Local backups use rsync as the backend, as it provides better control over local file permissions and can utilize hardlinks to provide a "more complete" snapshot of the source directory on a given day.

./goblin-store backup local -s <SOURCE_DIRECTORY> -d <DESTINATION_DIRECTORY>

This will provide the following directory structure:

DESTINATION_DIRECTORY/
├── 2026-09-03_13-14-07/          <- Baseline backup
│   ├── dir1/                     <- Real directory
│   │   └── data.bin              <- Original file (inode: 1001)
│   ├── dir2/
│   └── file1.txt                 <- Original file (inode: 1002)
│
└── 2026-09-04_00-00-00/          <- Next snapshot (hardlinked against previous)
    ├── dir1/                     <- Real directory (recreated)
    │   └── data.bin              <- Unchanged: hard link to previous run (inode: 1001, 0 bytes used)
    ├── dir2/
    ├── file1.txt                 <- Unchanged: hard link to previous run (inode: 1002, 0 bytes used)
    └── file2.txt                 <- New file copied from source (inode: 1003)

Breaking down the above command:

• ./goblin-store - running the app binary
• backup - run the backup command
• local - run the local subcommand
• -s <SOURCE_DIRECTORY> - the directory you'd wish to backup (REQUIRED)
• -d <DESTINATION_DIRECTORY> - the directory you'd wish to store the backups in (REQUIRED)

You may modify the behavior with these additional flags:

• -v - enable verbose console logging

Cloud Backups

Note: Cloud backends are currently not integrated into the app, but they can be used currently if you prefer the latest/incrementals directory structure on local disk.

Cloud backups utilize rclone as the backend for file copies, as it provides better support for cloud backends (S3, etc) and multi-threading support.

./goblin-store backup cloud -s <SOURCE_DIRECTORY> -d <DESTINATION_DIRECTORY>

This will provide the following directory structure:

DESTINATION_DIRECTORY/
├── 2026-09-03_13-14-07/          <- Baseline backup
│   ├── dir1/                     <- Real directory
│   │   └── data.bin              <- Original file (inode: 1001)
│   ├── dir2/
│   └── file1.txt                 <- Original file (inode: 1002)
│
└── 2026-09-04_00-00-00/          <- Next snapshot (hardlinked against previous)
    ├── dir1/                     <- Real directory (recreated)
    │   └── data.bin              <- Unchanged: hard link to previous run (inode: 1001, 0 bytes used)
    ├── dir2/
    ├── file1.txt                 <- Unchanged: hard link to previous run (inode: 1002, 0 bytes used)
    └── file2.txt                 <- New file copied from source (inode: 1003)

Breaking down the above command:

• ./goblin-store - running the app binary
• backup - run the backup command
• cloud - run the cloud subcommand
• -s <SOURCE_DIRECTORY> - the directory you'd wish to backup (REQUIRED)
• -d <DESTINATION_DIRECTORY> - the directory you'd wish to store the backups in (REQUIRED)

You may modify the behavior with these additional flags:

• -w <NUMBER> - the number of worker threads to use to backup (default is 4)
• -v - enable verbose console logging

Development

Requirements

• Go 1.25

Building

You can build Goblin Store using go build:

go build

And then run the binary:

./goblin-store backup ...

License

This project is available under the BSD 3-Clause license.