ad3laid3/disco-reaper
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

add docs

Browse source
This commit is contained in:
rambros 2026-03-21 19:43:39 +05:30
parent c4a6e18e2b
commit 11b9230078
5 changed files with 231 additions and 375 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

375
BACKUP.md
View file

@ -1,375 +0,0 @@
# Discord Reaper: Backup System Technical Specification
This document provides a deep-dive into the architecture, data lifecycle, and resilience strategies of the Discord Reaper backup system.
## 1. Architectural Overview
The backup system is built on a decoupled architecture that separates the API communication layer from the business logic and I/O operations.
- **`DiscordReader` (API Provider)**: A high-level wrapper around the `discord.py` library. It handles authentication, rate limiting, and provides an asynchronous interface for fetching guild data, message history, and binary assets. It focuses on *fetching* rather than *processing*.
- **`DiscordExporter` (Orchestration & Serialization)**: The core engine that defines the export lifecycle. It consumes data from the `Reader`, transforms it into standardized schemas, and manages local filesystem operations.
### Component Interaction Diagram
```mermaid
graph TD
A[UI / CLI] --> B[DiscordExporter]
B --> C[DiscordReader]
C --> D[Discord API]
B --> E[Local Filesystem]
B --> F[User Cache Object]
```
### File Tree Structure
```
DISCORD_BACKUP-{ServerID}/
├── server_profile/
│ ├── profile.json # Server metadata (name, ID, icon/banner paths)
│ ├── roles.json # All server roles (permissions, colors, positions)
│ ├── structure.json # Full category and channel hierarchy
│ ├── assets.json # Index of custom emojis and stickers
│ └── assets/ # Binary media files
│ ├── server_icon.png
│ ├── server_banner.png
│ ├── emoji_{name}_{id}.png
│ └── sticker_{name}_{id}.png
└── message_backup/
├── users/
│ ├── user_info.json # Deduplicated user profile cache
│ └── avatars/ # User avatar images
│ └── {user_id}.png
└── {channel_id}/
├── messages.json # Channel message history + metadata
├── attachments/ # Channel-level attachments
│ └── {filename}-{id_last_5}.{ext}
└── {thread_id}/ # Thread nested inside parent channel
├── thread_messages.json
└── thread_attachments/
└── {filename}-{id_last_5}.{ext}
```
---
## 2. Data Lifecycle & Serialization
### 2.1 Incremental Synchronization Algorithm
To achieve idempotency and efficiency, the system implements an incremental sync strategy using Discord's snowflake IDs.
1. **State Loading**: The `Exporter` reads the existing `{channel_id}/messages.json` (if present).
2. **Snowflake Extraction**: It extracts the `lastMessageID` from the metadata.
3. **Filtered Fetch**: It calls `fetch_message_history(after_id=last_id)`.
4. **In-Memory Merge**: New messages are appended to the existing list.
5. **Atomic Write**: The updated JSON is written back to disk, ensuring that only new delta data is fetched from the API.
### 2.2 User Profile Deduplication (`user_info.json`)
The system avoids redundant storage of user metadata (usernames, roles, colors) by using a global `user_cache` map.
- **Key**: `userID` (Snowflake).
- **Policy**: Users are added to the cache only on their first appearance in any channel's history.
- **Avatar Persistence**: User avatars are stored in `message_backup/users/avatars/` and referenced by relative paths in the JSON schemas.
---
## 3. Special Channel Type Specifications
### 3.1 Forum Channels & Threads
Forums present a hierarchical challenge where the "starter message" and the "conversation" exist in separate contexts.
- **Forum Index (`{forum_id}/messages.json`)**: Contains an enriched list of "starter messages" representing each thread. These entries include thread titles, applied tags, and total attachment stats (summed from the entire thread).
- **Thread Persistence**: All threads nest inside their parent channel directory:
- **Forum Threads**: `message_backup/{forum_id}/{thread_id}/thread_messages.json`
- **Regular Threads**: `message_backup/{parent_channel_id}/{thread_id}/thread_messages.json`
- **Starter Identification**: The system uses `thread.history(limit=1, after=snowflake(thread_id - 1))` to reliably capture the first post even if it has been edited or pinned.
---
## 4. Resilience & Error Handling
### 4.1 Permission Resilience (403 Forbidden)
The system is designed to "fail-soft" when encountering restricted content:
- **Server Level**: If the bot lacks `view_channel` or `read_message_history` globally, the backup aborts with a clear error.
- **Channel Level**: If a specific channel is restricted, the error is logged, and the system proceeds to the next channel to ensure a partial backup is still completed.
- **Asset Level**: If an emoji or sticker cannot be downloaded due to permissions, the metadata is preserved with a `null` local path.
### 4.2 Lottie Sticker Workaround
Discord's Lottie stickers (format: 3) are not supported by standard `discord.py` save methods. The system implements a bypass:
1. Extracts the internal `aiohttp` session from the client: `client.http._HTTPClient__session`.
2. Performs a direct `GET` request to the sticker URL.
3. Streams the raw byte data directly to a `.json` file locally.
---
## 5. Technical Schemas
### 5.1 Message Object (`_format_message`)
The internal representation of a message focuses on portability:
| Field | Type | Description |
| :--- | :--- | :--- |
| `messageID` | `String` | Original Discord Snowflake |
| `type` | `String` | Normalized type (Text, ThreadStarter, Forward, etc.) |
| `timestamp` | `ISO8601` | Created date/time |
| `isPinned` | `Boolean` | Pin status |
| `content` | `String` | Raw markdown content (or snapshot content for forwards) |
| `userID` | `String` | Reference to `user_info.json` |
| `attachments`| `Array` | List of local file references and metadata |
| `embeds` | `Array` | Raw Discord-formatted embed objects |
| `stickers` | `Array` | List of Message Sticker objects (see below) |
| `reactions` | `Array` | List of Reaction objects |
#### Message Sticker Object
| Field | Type | Description |
| :--- | :--- | :--- |
| `id` | `String` | Sticker Snowflake ID |
| `name` | `String` | Sticker name |
| `format` | `String` | File format (PNG, APNG, LOTTIE, GIF) |
| `localPath` | `String` | Relative path to local file in `{channel_id}/attachments/` |
#### Reaction Object
| Field | Type | Description |
| :--- | :--- | :--- |
| `emoji` | `String` | String representation (`unicode` or `name:id`) |
| `count` | `Integer` | Total count of this reaction |
### 5.2 Asset Naming Logic
To prevent filename collisions (e.g., multiple files named `image.png`), the system uses a suffixing strategy:
`{filename_stem}-{snowflake_last_5}.{ext}`
Example: `sunset-54321.png`
### 5.3 `profile.json` Specification
Path: `server_profile/profile.json`
| Field | Type | Description |
| :--- | :--- | :--- |
| `name` | `String` | Original Discord guild name |
| `id` | `String` | Guild Snowflake ID |
| `icon` | `String` | Relative path to local guild icon in `server_profile/assets/` |
| `banner` | `String` | Relative path to local guild banner in `server_profile/assets/` |
| `last_backup` | `ISO8601` | Timestamp of the last successful backup run |
| `ignore_channels` | `Array` | List of channel Snowflakes explicitly excluded from backup |
### 5.4 `roles.json` Specification (Array of objects)
Path: `server_profile/roles.json`
| Field | Type | Description |
| :--- | :--- | :--- |
| `id` | `String` | Role Snowflake ID |
| `name` | `String` | Role name |
| `color` | `String` | Hex-string representation of role color (e.g. `"#ffffff"`) |
| `position` | `Integer` | Vertical position in the hierarchy (0 is bottom) |
| `permissions`| `Integer` | Bitwise integer representing the role's Discord permissions |
| `hoist` | `Boolean` | Whether the role is displayed separately in the sidebar |
| `mentionable`| `Boolean` | Whether the role can be mentioned |
### 5.5 `assets.json` Specification
Path: `server_profile/assets.json`
Contains two primary arrays: `emojis` and `stickers`.
#### Emoji Object
| Field | Type | Description |
| :--- | :--- | :--- |
| `id` | `String` | Emoji Snowflake ID |
| `name` | `String` | Emoji name (without colons) |
| `animated` | `Boolean` | True if the emoji is a GIF |
| `filename` | `String` | Filename within `server_profile/assets/` |
#### Sticker Object
| Field | Type | Description |
| :--- | :--- | :--- |
| `id` | `String` | Sticker Snowflake ID |
| `name` | `String` | Sticker name |
| `filename` | `String` | Filename within `server_profile/assets/` |
### 5.6 `structure.json` Specification (Array of Category objects)
Path: `server_profile/structure.json`
#### Category Object
| Field | Type | Description |
| :--- | :--- | :--- |
| `type` | `String` | Always `"category"` |
| `id` | `String` | Category Snowflake ID (or `"uncategorized"`) |
| `name` | `String` | Category name |
| `position` | `Integer` | Vertical position in hierarchy |
| `channels` | `Array` | List of Channel objects (see below) |
#### Channel Object
| Field | Type | Description |
| :--- | :--- | :--- |
| `id` | `String` | Channel Snowflake ID |
| `name` | `String` | Channel name |
| `type` | `String` | "text", "voice", "forum", "news", or "thread" |
| `position` | `Integer` | Vertical position within the category |
| `topic` | `String` | Channel description/topic (null if empty) |
| `nsfw` | `Boolean` | True if marked Restricted/NSFW |
| `available_tags` | `Array` | List of Forum Tag objects (see below) |
#### Forum Tag Object
| Field | Type | Description |
| :--- | :--- | :--- |
| `id` | `String` | Tag Snowflake ID |
| `name` | `String` | Tag display name |
| `moderated` | `Boolean` | True if restricted to moderators |
| `emoji_id` | `String` | ID of the tag's emoji (null if unicode/none) |
| `emoji_name` | `String` | Name of the tag's emoji |
### 5.7 `user_info.json` Specification (Array of User objects)
Path: `message_backup/users/user_info.json`
| Field | Type | Description |
| :--- | :--- | :--- |
| `userID` | `String` | User Snowflake ID |
| `username` | `String` | Current global username |
| `userNickname`| `String` | Server-specific nickname (display name) |
| `userColor` | `String` | Role-derived color for the user |
| `userIsBot` | `Boolean` | True if the account is a bot |
| `userRoles` | `Array` | List of role snippets (name, id, color, position) |
| `userAvatar` | `String` | Relative path to local avatar in `users/avatars/` |
### 5.8 Channel History JSON Specification
Path: `message_backup/{channel_id}/messages.json`
This file contains the full history of a channel along with synchronization metadata.
| Field | Type | Description |
| :--- | :--- | :--- |
| `channelName` | `String` | Human-readable name of the channel |
| `channelID` | `String` | Channel Snowflake ID |
| `channelType` | `String` | "Text", "Thread", "News", or "Forum" |
| `messageCount` | `Integer` | Total number of messages stored in the `messages` array |
| `threadCount` | `Integer` | (If Parent) Count of threads associated with this channel |
| `lastMessageID`| `String` | ID of the most recent message (used for incremental sync) |
| `totalAttachmentSizeBytes`| `Integer`| Summed size of all attachments for this channel |
| `numberOfAttachments` | `Integer`| Total count of attachments |
| `lastBackup` | `ISO8601` | Timestamp of last message fetch |
| `messages` | `Array` | The message objects (see Section 5.1) |
| `parentID` | `String` | (If Thread) Snowflake of the parent channel |
### 5.9 Thread History JSON Specification
Path: `message_backup/{channel_id}/{thread_id}/thread_messages.json`
Same schema as Section 5.8, with `channelType` set to `"Thread"` and `parentID` always present.
---
## 7. Backup Reader Implementation Guide
This section is a technical manual for developers building third-party tools (viewers, search engines, or analytics) to consume Discord Reaper backups.
### 7.1 Entry Point Discovery
A reader should start by identifying the backup root directory (prefixed with `DISCORD_BACKUP-`).
1. **Parse `server_profile/profile.json`**: Extract the server name, ID, and assets (icon/banner).
2. **Load `server_profile/structure.json`**: This defines the navigation tree for your UI.
- Iterate through categories.
- Map channels to their respective types (text, voice, forum).
- Store the `position` to preserve the original visual order.
### 7.2 Relational Data Mapping
The backup data is normalized to minimize duplication. A reader must implement the following resolve logic:
- **User Resolution**: When parsing a message in `{channel_id}/messages.json`, the `userID` must be cross-referenced against the `userID` keys in `message_backup/users/user_info.json`.
- **Role Resolution**: Use the `userRoles` array (IDs) from the user object and resolve them against the role metadata in `server_profile/roles.json` to get colors and names.
- **Static Asset Resolution**:
- **Server Assets**: Prepend `server_profile/assets/` to filenames found in `server_profile/assets.json`.
- **User Avatars**: Resolve `userAvatar` paths found in `user_info.json` (pointing to `users/avatars/`).
### 7.3 Message Rendering Logic
When rendering the `messages` array from a channel JSON:
| Feature | Reader Implementation Logic |
| :--- | :--- |
| **Markdown** | Content is raw Discord markdown. Use a library like `markdown-it` with discord-specific plugins. |
| **Attachments** | Resolve `url` field (`{channel_id}/attachments/{filename}`) relative to the `message_backup/` directory. |
| **Emojis/Stickers** | If a message contains custom emojis/stickers, resolve their metadata via `server_profile/assets.json`. |
| **Replies** | Use the `reference` object to find the target `messageId`. Note: The target might be in the same file or a different channel/thread. |
### 7.4 Thread & Forum Reconstruction
Reconstructing the hierarchy requires specific pointer logic:
1. **Forums**:
- Read `message_backup/{forum_id}/messages.json`.
- Each message in this file is a `ThreadStarter`.
- The `messageID` of the starter message *is usually* the same as the `thread_id`.
- To load the full thread, open `message_backup/{forum_id}/{thread_id}/thread_messages.json`.
2. **Regular Threads**:
- Discoverable via the `parentID` field in any message or by scanning for `thread_messages.json` inside channel directories.
- Match the `thread.id` in a `ThreadStarter` message to the respective subdirectory.
---
## 8. Discord.py Model Hydration Guide
If you are building a `discord.py` API-compatible wrapper to read these backups directly into familiar Discord objects, here is the explicit property mapping from the schema to the standard `discord.py` object attributes.
### 8.1 Base Server (Guild)
File: `server_profile/profile.json` & `server_profile/roles.json` & `server_profile/structure.json`
- **`discord.Guild`**:
- `id`: Cast `id` (str) to `int`.
- `name`: Mapped directly from `name`.
- `icon` / `banner`: Represented as `discord.Asset` objects. Use the local file paths from `icon` / `banner` as the asset URL/filepath.
- `roles`: Hydrated from `server_profile/roles.json`.
- `channels` / `categories`: Hydrated from `server_profile/structure.json`.
### 8.2 Roles (`discord.Role`)
File: `server_profile/roles.json`
- `id`: Cast `id` to `int`.
- `name`: Mapped directly.
- `color`: Parse the hex string to `discord.Color(value)`.
- `position`: Mapped directly.
- `permissions`: Initialize `discord.Permissions(value=int(permissions))`.
- `hoist`: Mapped directly to boolean.
- `mentionable`: Mapped directly to boolean.
### 8.3 Users & Members (`discord.Member` / `discord.User`)
File: `message_backup/users/user_info.json`
- `id`: Cast `userID` to `int`.
- `name`: Mapped from `username`.
- `display_name`: Mapped from `userNickname`.
- `bot`: Mapped from `userIsBot`.
- `color`: Parse `userColor` string to `discord.Color`.
- `roles`: List of hydrated `discord.Role` objects via matching `id`s from the `userRoles` array.
- `avatar`: Mocked `discord.Asset` using the `userAvatar` local path.
### 8.4 Channels (`discord.TextChannel`, `discord.CategoryChannel`, `discord.ForumChannel`)
File: `server_profile/structure.json`
- Iterate over the top-level array (Categories):
- **`discord.CategoryChannel`**:
- `id`: Cast `id` to `int`.
- `name`: Mapped directly.
- `position`: Mapped directly.
- Iterate over the nested `channels` array:
- **`discord.abc.GuildChannel` classes**:
- `id`: Cast `id` to `int`.
- `name`: Mapped directly.
- `position`: Mapped directly.
- `type`: Match the `type` string back to the `discord.ChannelType` enum.
- `category_id`: Inherited from the parent category block.
- `topic`: Mapped directly (if applicable).
- `nsfw`: Mapped directly to boolean.
### 8.5 Messages (`discord.Message`)
File: `message_backup/{channel_id}/messages.json` (Iterating the `messages` array)
- `id`: Cast `messageID` to `int`.
- `type`: Map the string `type` (e.g., "Default", "Reply") to `discord.MessageType`.
- `created_at`: Parse `timestamp` (ISO-8601 string) into a timezone-aware `datetime` object.
- `pinned`: Mapped from `isPinned`.
- `content`: Mapped from `content`.
- `author`: Resolve the `userID` against the loaded `discord.Member` mocks.
- `embeds`: Instantiate using `discord.Embed.from_dict(embed_dict)` directly on the elements of the `embeds` array.
- **Reference (Replies)**:
- If `reference` exists, hydrate a `discord.MessageReference`.
- `message_id`: Cast `reference.messageId` to `int`.
- `channel_id`: Cast `reference.channelId` to `int`.
### 8.6 Attachments (`discord.Attachment`)
Nested within Message objects.
- `id`: Cast `id` to `int`.
- `filename`: Mapped from `fileName`.
- `size`: Mapped from `fileSizeBytes`.
- `url` / `proxy_url`: Point to the local relative path (`{channel_id}/attachments/{resolved_filename}`).
### 8.7 Reactions (`discord.Reaction` & `discord.PartialEmoji`)
Nested within Message objects.
- `count`: Mapped from `count`.
- `emoji`: Iterate the `emoji` string. If custom (contains a `:`), split it to mock a `discord.PartialEmoji(name=..., id=...)`. Otherwise, mock standard unicode strings.

62
docs/backup-specs.md Normal file
View file

@ -0,0 +1,62 @@
# Discord Reaper: Backup System Technical Specification
Discord Reaper uses a high-performance **SQLite-based** backup system designed to handle massive community snapshots with millions of messages, deduplicated media, and cross-session persistence.
---
## 1. Architectural Overview
The backup system has transitioned from a legacy JSON-based flat-file structure to a central **SQLite Database** (`reaper.db`) architecture. This allows for:
- **O(1) lookups**: Instant mapping of original message IDs to target message IDs.
- **Memory Efficiency**: The system no longer loads massive message lists into RAM; it streams data directly from disk.
### Component Relationship
```mermaid
graph TD
A[Discord API] --> B[DiscordReader]
B --> C[BackupDatabase]
C --> D[reaper.db]
C --> E[Media Pool /cas/]
D --> F[Migration Shuttle]
```
---
## 2. The Database Schema (`reaper.db`)
The backup is stored in a single SQLite file, typically found in your `ReaperFiles-{ServerID}/` directory. The schema is normalized to prevent redundancy.
### Core Tables
* **`guild_profile`**: Stores server name, ID, description, owner, and icon/banner URLs.
* **`roles` & `permissions`**: Captures every custom role (colors, bits) and complex channel-specific permission overwrites.
* **`channels` & `threads`**: Detailed metadata including category nesting, NSFW flags, bitrates, and thread archive statuses.
* **`messages`**: The central history table. Stores sender IDs, timestamps, content, and references.
* **`attachments` / `embeds` / `reactions`**: Relational tables linked to `messages` for storing rich data.
* **`users`**: A deduplicated author cache (usernames, avatars, server roles).
* **`user_alias`**: Stores mapping between User IDs and their generated "Privacy Aliases" (e.g., `SwiftFox`).
---
## 3. Media Pool & CAS Logic
Reaper implements a **Content-Addressable Storage (CAS)** system for all media (images, videos, stickers, avatars).
### How it works:
1. **Hashing**: When a file is downloaded, Reaper calculates its **SHA-256 hash**.
2. **Deduplication**: If another message contains the exact same image (even years later or in a different channel), Reaper sees the hash already exists in the `media_pool` table.
3. **Referencing**: Instead of downloading the file again, Reaper simply creates a reference to the existing file in the `/cas/` directory.
**Benefit**: This significantly reduces the disk footprint of backups for servers where users frequently share the same memes or assets.
---
## 4. Incremental Synchronization
Reaper uses the **Snowflake ID** logic to perform "Delta Backups":
1. **Scan**: The system queries the `messages` table for the highest (newest) ID in a specific channel.
2. **Fetch**: It then calls the Discord API to fetch messages `after` that specific ID.
3. **Merge**: Only the brand-new messages are inserted into the database.
This makes "keeping a backup updated" an extremely fast operation, even for servers with millions of existing messages.
---

47
docs/faq.md Normal file
View file

@ -0,0 +1,47 @@
# DiscoReaper - Help & FAQ
### Ctrl+V doesnt work for pasting the tokens?
- You can try pasting with `Ctrl+Shift+V`, this is also the default keybind for linux systems.
- Windows 10 has been known to be problematic, it seems to use a separate clipboard for the terminal.
- As a workaround, you can directly paste tokens in **config.yaml** file inside the ReaperFiles-XXX folder
```
discord_bot_token: YOUR_DISCORD_BOT_TOKEN
fluxer_bot_token: YOUR_FLUXER_BOT_TOKEN
stoat_bot_token: YOUR_STOAT_BOT_TOKEN
```
### Messages are empty, Only Timestamps are migrated?
- The Bot might be missing the **Message Content Intent** Privilege.
- **Enable it** in the Discord Developer Portal under the **Bot** tab.
### Why are some Discord channels are missing?
- If the missing channel has any custom permission overrides, **add the bot role manually** to the **channel or its parent category**, this will allow the bot to access them.
- The bot role may not have access to the private channels in your discord server, Even when you give Read Messages & View Channels permission to the bot role.
---
### Can I use the tool to delete messages in the Discord server?
- **No.** The tool operates using the Discord Bot API with **read-only permissions** (view and read access).
- It does not perform any write actions, so it cannot modify, delete, or change anything in your Discord server.
### Can I migrate Personal messages (DMs)?
- **No.** The tool uses the Discord Bot API, which does not grant access to personal messages (DMs), so they cannot be migrated or exported.
### Which Platforms are supported?
- **Fluxer** and **Stoat** are the supported target platforms.
- Eligibility criteria for new platforms:
- Open Source
- Self-Hostable
- Bot API
---
### Where do I get help?
Ping me in the [Reaper Community](https://fluxer.gg/9KxDP8WH) on Fluxer
Provide the following details when asking for help:
- Your Operating System
- Reaper Version
- Briefly describe your issue

85
docs/features.md Normal file
View file

@ -0,0 +1,85 @@
# Extensive Feature list
Here's complete list of everything the tool can handle and any specific caveats or workarounds.
- 🟩Fully supported
- 🟧Work Around implemented
- 🟥Limitation
---
# Migration Tool
## Server Identity
* 🟩 **Server Name**
* 🟩 **Server Icon**
* 🟩 **Server Banner**
## Server Structure
* 🟩 **Categories**
- 🟩 **Metadata**: Copies Category names, also preserve the position of the Categories and its child channels.
- 🟩 **Permission Overwrites**: Copies role-specific permission overwrites set at the category level.
* 🟩 **Channels**
- 🟩 **Metadata**: Copies Channel names, topics/descriptions.
- 🟩 **NSFW Flag** is preserved for age restricted channels.
- 🟧 **Slowmode** timing rules are preserved for text channels (Stoat doesnt have slowmode).
- 🟩 **Bitrate** setting is also preserved Voice channels
- 🟩 **Permission Overwrites**: Copies role-specific permission overwrites set on individual channels.
## Roles
* 🟩 **Role Cloning**: Clones every custom role with its original name, color (hex code), and hierarchical position.
* 🟩 **Role Settings**: Copies the "Display role members separately from online members" (hoist) and "Allow anyone to @mention this role" settings.
### Text Messages
* 🟩 **Markdown Support**: Preserves all standard and advanced Discord markdown:
- `**Bold**`, `*Italics*`, `__Underline__`, `~~Strikethrough~~`.
- `> Blockquotes` and `>>> Multi-line quotes`.
- `Code blocks` and ` ```Full code blocks with syntax highlighting``` `.
- `||Spoilers||`.
* 🟩 **Message Authors**: Preserves the original author's name and avatar in migrated messages.
* 🟩 **Reply Links**: Preserves the connection of replies if the referenced message was previously migrated.
* 🟩 **Embeds**: Clones rich embeds sent by bots or webhooks, including their colors, fields, and descriptions.
### File Attachments
* 🟩 **Media**: Downloads and re-uploads images, videos, and audio clips.
* 🟩 **Documents**: Preserves PDFs, text files, and other message attachments.
* 🟧 **Size Limits**: Automatically checks size limits. If an attachment is too large for the target platform.
> Stoat doesnt support many file types, so those attachments are skipped and an error message will be sent in place of the attachment
### Emojis
* 🟩 **Standard Emojis**: Full support for Unicode emojis.
* 🟩 **Custom Server Emojis**: Automatically clones custom server emojis (both static and animated) to the target server.
* 🟩 **Emoji Mapping in messages**: References to custom emojis in migrated messages are updated to point to the new emoji IDs on the target platform.
* 🟥 **External Emojis**: Emojis from external servers are not migrated; only the `:emoji_name:` will be displayed.
### Stickers
* 🟧 **Normal Stickers**: Migrates static and animated stickers (`.png`, `.apng`, `.gif`).
* 🟧 **Lottie Stickers**: Converted during migration:
- **Fluxer**: Converted to `.webp` for compatibility.
- **Stoat**: Converted to `.gif`.
> Stickers are currently migrated as attachments since sending stickers via api is not yet fully supported on target platforms.
### Mentions & Discord Links
* 🟩 **Channel Mentions**: Automatically resolves `#channel` mentions to point to the correct migrated channel on the target server.
* 🟩 **Role Mentions**: Automatically resolves `@role` mentions to point to the cloned role on the target platform.
* 🟩 **User Mentions**: Mentions of users are preserved as text [`@nickname`] if that user.
* 🟩 **Message Links**: Standard Discord message URLs are resolved to the new URL on Fluxer or Stoat (as long as the target message has been migrated).
* 🟩 **Channel Links**: Discord channel URLs (like `discord.com/channels/...`) are resolved to their matching target channel link.
### Threads & Forums
* 🟩 **Active & Archived Threads**: Scans and migrates messages from all threads.
* 🟧 **Thread Structure**: Maintains the nesting of threads within their parent channels.
> This is a workaround as Threads and Forums are not yet implement in Fluxer and Stoat
### Misc features
* 🟩 **Incremental Sync**: Scans migration database to find the "Last Message ID." It skips duplicate data and continues where it left off.
* 🟩 **Audit Logging**: Creates a `#reaper-logs` channel in the target community and sends status updates and error reports.
---
# Backup Tool
You can create a local snapshot of your Discord server using the Backup Tool.
* 🟩 **SQLite Database**: All server mappings, message IDs, and content are stored in a SQLite database.
* 🟩 **Deduplicated Attachments**: All the attachment media is stored in a content-addressable storage system.
* 🟩 **Incremental Syncing**: Scans your existing local files and only downloads messages that have been sent since your last backup.
* 🟩 **Supported Migrated**: Local backup can be used to later migrate your data without any dependance on discord API - even if that server was deleted.

37
docs/guide.md Normal file
View file

@ -0,0 +1,37 @@
# Discord Reaper Operations
## 1. Migration & Shuttle Tools
The primary suite of tools for moving data between Discord & Fluxer/Stoat.
* **Shuttle Migration (Direct)**:
Connects directly to Discord and your target platform to migrate categories, channels, roles, and messages in real-time. It uses intelligent mapping to ensure links and mentions work in the new community.
* **Resumed Migration**:
If a large migration is interrupted or hits a rate limit, the Resumed Migration tool uses the local SQLite database to pick up exactly where it left off, skipping messages that have already been transferred.
* **Offline Migration (from Backup)**:
Allows you to migrate data without reaching out to Discord's servers. It uses a previously created local backup as the source, making it the safest and fastest way to migrate if the original Discord server has already been deleted or restricted.
* **Batch Channel Processing**:
Allows you to select up to 50 specific channels or categories to migrate at once. Reaper will process them sequentially to stay within platform rate limits.
---
## 2. Backup & Archiving
Tools for creating and maintaining permanent local copies of your data.
* **Full Server Backup**:
Downloads the entire server structure-including all categories, channels, custom roles, and permission overwrites-along with full message histories and all file attachments into a high-performance SQLite database.
* **Incremental Syncing**:
Scans your existing local backup database and only downloads messages sent since the last run. This is extremely efficient for keeping a continuous archive of active communities.
* **Media Deduplication**:
An background operation that ensures identical file attachments (images, stickers, etc.) are only stored once on your hard drive, even if they appear in multiple messages or channels.
---
## 3. Danger Zone (Advanced Management)
Powerful tools for server cleanup and reset. Use with caution.
* **Server Wipe**:
A high-speed utility that deletes all categories, channels, and custom roles on a target server. This is used to "reset" a community before testing a fresh migration. Requires explicit confirmation.
* **Permission Purge**:
Scans every channel and category on the target server and removes all role and user-specific permission overwrites, restoring the server to its default permission state.
---