dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
af327bcc5ef036fd47f839a9d9831d067dda0f09
serverAppConfigs/serverAppConfigs-docs/Dankserver/RomM-Scraping.md
T
Frank a5fe045b58 docs(vault): one fact, one owner — repair the doubled pages, single-source the facts 
Structural repair:
- 13 pages were the same document twice (each exactly 2x its HEAD size),
  a second frontmatter block whose closing fence had been glued onto a body
  line by a frontmatter rewrite that re-read its own output. Un-doubled by
  cutting at the second frontmatter and keeping the copy whose body is a
  superset; the orphan census reports 0 lost tokens against HEAD.
- log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first —
  66 dated headings merged into 34, titles demoted to bold leads, every
  content line verified present. The file had been descending at the head
  and ascending at the tail.

Ownership moves:
- New `LLM/frank-medium.md` is the locator for the retired Dankserver
  endpoint: era windows quoted from each era page's own banner, no numbers
  restated, and the two unresolved date conflicts recorded instead of
  silently resolved. `frank-archon.md` links to it instead of restating the
  pre-rename window.
- The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone;
  `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor
  verdicts and cross-split spec decode, and the four links that pointed at
  the stubs were repointed at it or at the drafter page.
- Authentik's three stub pages fold into `Authentik.md`, with the process
  detail and the outpost reference split out so the parent stays under the
  length hint.

Verified corrections (host census + live probe, 2026-10-19):
- The MXFP4 checkpoint two pages called the "on-disk rollback" is gone:
  /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on
  the era page that owns that payload; the swap page now links to it.
- frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2
  arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run
  census is kept once (155 requests) instead of twice (~120 and 155).
- Fleet shapes re-confirmed against the live endpoints before touching any
  page: 3x200000 fp8 KV, 2x200704, 2x262144.

Index rows are regenerated, not hand-carved. Net -1,849 vault lines.
2026-10-19 05:00:43 -07:00

306 lines
11 KiB
Markdown
Raw Blame History

---
type: reference
description: "Metadata providers, cover art, ROM naming and the scraping workflow behind the RomM library."
tags:
- dankserver
- romm
timestamp: 2026-08-28
---
# RomM Scraping & Asset Management
This guide covers metadata providers, cover art, ROM naming conventions, emulator integration, and the scraping workflow for RomM on Dankserver.
**Reference:** https://docs.romm.app
## 1. Metadata Providers
RomM v5.1.0+ supports 11+ metadata providers. The `scan.priority.metadata` list determines which provider wins for descriptive fields. The `scan.priority.artwork` list controls cover art sourcing separately.
### Screenscraper (ScreenScraper.fr)
- **Type:** Free, community-driven (French provider)
- **Credentials:** `SCREENSCRAPER_USER` + `SCREENSCRAPER_PASSWORD` (account user/pass)
- **Registration:** https://www.screenscraper.fr/membreinscription.php
- **Rate limits:** No documented hard cap. Community-claimed ~5 req/sec burst.
- **Capabilities:** Titles, descriptions, cover art, screenshots, manuals, 3D boxes, CD/cartridge covers, marquee logos, fanart, bezels, gameplay videos.
- **Region support:** Excellent. Respects `scan.priority.region` and `scan.priority.region_mode`.
- **Best for:** Richest media coverage (all art types), multi-platform, free. Use as secondary to IGDB for metadata, or swap priority order if Screenscraper art quality is preferred.
### IGDB (Internet Game Database / Twitch)
- **Type:** Freemium (free tier with limits; paid for higher volume)
- **Credentials:** `IGDB_CLIENT_ID` + `IGDB_CLIENT_SECRET` (Twitch OAuth2)
- **Registration:** Twitch Developer Portal. Create app with name `romm-<random hash>`, OAuth redirect `localhost`, category `Application Integration`, client type `Confidential`.
- **Rate limits:** ~10 req/sec for free tier. Cached responses reduce effective calls.
- **Capabilities:** Titles, descriptions, cover art, screenshots, related games, release dates, genres, platforms.
- **Limitations:** No 3D boxes or physical media art. No manuals. No marquee/fanart.
- **Default priority:** Highest (`igdb` listed first in both metadata and artwork priority).
### MobyGames
- **Type:** Paid (hobbyist API subscription; non-commercial use only)
- **Credentials:** `MOBYGAMES_API_KEY`
- **Rate limits:** Hobbyist tier ~0.2 req/sec (one request every 5 seconds).
- **RomM recommendation:** "We recommend using ScreenScraper instead, as it is free to use."
- **Best for:** Supplemental metadata depth for older PC/console titles.
### RetroAchievements (RAHasher)
- **Type:** Free, hash-based achievement tracking
- **Credentials:** `RETROACHIEVEMENTS_API_KEY`
- **How it works:** RomM bundles RAHasher (v1.8.3+). ROMs are hashed during scans, matched against RA online database. Achievement progress synced per-user via their RA username.
- **Known issues:** Folder-based disc ROMs (`.cue` + `.bin`, `.gdi`) may fail hashing. Non-zip archives (`.7z`, `.rar`) had incorrect hash computation. Ensure `filesystem.skip_hash_calculation` is `false`.
- **Important:** Achievements cannot be earned via RomM's built-in EmulatorJS player. Must use an RA-capable standalone emulator.
### Additional Built-in Providers
| Provider | Slug | Enabled Via | Type |
|----------|------|-------------|------|
| Hasheous | `hasheous` | `HASHEOUS_API_ENABLED=true` | Free, hash-based, proxies IGDB data |
| Playmatch | `playmatch` | `PLAYMATCH_API_ENABLED=true` | Multi-provider hash matching |
| LaunchBox | `launchbox` | `LAUNCHBOX_API_ENABLED=true` + XML | Local XML database |
| Flashpoint | `flashpoint` | `FLASHPOINT_API_ENABLED=true` | 180,000+ browser games |
| HowLongToBeat | `hltb` | `HLTB_API_ENABLED=true` | Game completion times |
| SteamGridDB | `steamgriddb` | `STEAMGRIDDB_API_KEY` | Custom cover art |
| ES-DE gamelist | `gamelist` | Place `gamelist.xml` in platform folder | Local XML |
## 2. Recommended Provider Combinations
| Combo | Providers | Best For | Cost |
|-------|-----------|----------|------|
| Quick Starter | Hasheous only | No API keys needed | Free |
| French Connection | Screenscraper + RetroAchievements | Rich media, free | Free |
| Chef's Choice (recommended) | Hasheous + IGDB + SteamGridDB + RetroAchievements | Best overall quality | Free (IGDB free tier) |
| Twitch Fanboy | IGDB + Playmatch | Single-provider solution | Free |
**For Dankserver,** the recommended setup:
- `IGDB` (primary metadata) + `SCREENSCRAPER` (artwork backup + rich media) + `RETROACHIEVEMENTS` (achievements) + `HASHEOUS_API_ENABLED=true` (fast hash-based fallback)
## 3. ROM File Naming Conventions
### No-Intro Naming Scheme (Recommended)
The de facto standard for cartridge and handheld ROMs. Only `Title` and `Region` are required.
```
Super Metroid (Japan, USA) (En,Ja)
Sonic the Hedgehog (USA)
Metal Gear Solid (USA) (Disc 1)
[BIOS] PS1 System BIOS (USA)
Tetris [!].gba
```
**Field meanings:**
- `(Region)`: `USA`, `EUR`, `JPN`, `UE` (USA+Europe), `JUE` (Japan+USA+Europe)
- `(Languages)`: `En`, `Ja`, `De`, `Fr`, `Es`
- `[Status]`: `[!]` = verified good dump, `[b]` = bad dump, `[c]` = completed
- `(Devstatus)`: `(Proto)`, `(Demo)`, `(Release Candidate)`
- `[BIOS]` prefix: Identifies BIOS files
### RomM's Region/Language/Revision Parsing
RomM parses inline tags from filenames:
```
tetris (En) [1.0001] [HACK] [!].gba
```
- Region/language: ISO-like codes or full names
- Revision: `rev v1` or `rev-1`
- Provider ID tags: `(igdb-xxxx)`, `(ssfr-xxxx)`, `(moby-xxxx)`, `(ra-xxxx)` — forces specific provider match
### Best Practices for Dankserver
1. Use No-Intro naming as the standard.
2. Use **Igir** to rename and organize ROMs into RomM's required folder structure:
```
igir copy zip test clean \
--dat "No-Intro*.zip" \
--input "ROMs" \
--output "/opt/romm/roms/{romm}"
```
3. Keep `[BIOS]` prefix for BIOS files in the `bios/` directory.
4. One ROM file per game folder where possible.
5. Use archive formats (`.zip`, `.7z`) — RomM supports them natively.
## 4. Cover Art & Media Assets
### Media Types
| Media Type | Description | Best Provider |
|-----------|-------------|---------------|
| `box2d` | Standard 2D front cover art | IGDB, Screenscraper |
| `box3d` | 3D box art | Screenscraper |
| `miximage` | Composite (box + screenshot + logo) | Screenscraper |
| `screenshot` | In-game screenshot | IGDB, Screenscraper |
| `marquee` | Transparent logo/header | Screenscraper |
| `bezel` | EmulatorJS-compatible bezel | Screenscraper |
| `manual` | PDF instruction manual | Screenscraper |
| `video` | Gameplay video/trailer | Screenscraper |
### Custom Artwork Directories
RomM supports manual artwork placement via subdirectories within each game folder:
```
/opt/romm/library/roms/{platform}/{game}/
game.rom
covers/
screenshots/
3dboxes/
manuals/
fanart/
videos/
marquees/
bezels/
```
For ES-DE gamelist imports, media maps as:
| gamelist.xml Tag | Folder Name |
|-----------------|-------------|
| `<cover>` | `covers/` |
| `<backcover>` | `backcovers/` |
| `<box3d>` | `3dboxes/` |
| `<marquee>` | `marquees/` |
| `<screenshot>` | `screenshots/` |
| `<fanart>` | `fanart/` |
| `<manual>` | `manuals/` |
### MAME Artwork
MAME uses cabinet pics, flyers, marquees, control panels, backdrops, overlays. Stored in `{platform}/mame_artwork/{game}/` subdirectories.
Sources: Mr. Do's Arcade (https://www.mameworld.info/mrdo/), ProgettoSnaps (http://www.progettosnaps.net/artworks/).
## 5. Emulator Integration
### EmulatorJS (Built-in)
Web-based player using RetroArch cores compiled to WebAssembly. Runs in-browser with no local emulator needed. Configured via `emulatorjs.*` settings in `config.yml`. Supports netplay via STUN/TURN servers.
### Emulator Streaming
Launch native emulator containers (PCSX2, Dolphin, Xemu) and stream picture/sound/input back to the browser. Configured via `streaming.*` in `config.yml`.
### BIOS File Requirements
BIOS files placed in `/opt/romm/library/bios/{platform}/`. Platform subdirectories required. File names should match emulator expectations:
| Platform | BIOS File |
|----------|-----------|
| PlayStation | `scph1001.bin`, `scph5501.bin`, `scph5502.bin` |
| GB/GBC | `gb_bios.bin`, `gbc_bios.bin` |
| SNES | `sfc_bios.bin` |
BIOS matching done by hash against platform requirements.
### Folder Structure Options
**Structure A (recommended):**
```
/opt/romm/library/
roms/
└── {platform}/
└── {game}.rom
bios/
└── {platform}/
└── {bios_file}.bin
```
**Structure B (fallback):**
```
/opt/romm/library/
└── {platform}/
├── roms/
│ └── {game}.rom
└── bios/
└── {bios_file}.bin
```
Dankserver mounts `/opt/romm/library:/romm/library` — RomM auto-detects structure.
## 6. Scraping Workflow
### Recommended Pipeline
1. Organize ROMs → No-Intro naming + RomM folder structure.
2. Create `config.yml` → Set provider credentials, priority lists, region prefs.
3. First scan → Scan ALL → verify everything is matched.
4. Review unmatched → Manually match or add provider tags to filenames.
5. Fetch artwork → Re-scan with `scan.media` extended to desired types.
6. Sync RetroAchievements → Set user RA username → trigger sync.
7. Export gamelist → Enable `scan.gamelist.export` for ES-DE/Batocera.
8. Maintain → Enable scheduled rescans for new DAT updates.
### Scan Types
| Scan Type | What It Does |
|-----------|-------------|
| ALL | Full scan — hash, match metadata, fetch artwork |
| UNMATCHED | Only matches unmatched ROMs (faster) |
| METADATA | Updates metadata only |
| ARTWORK | Re-fetches artwork only |
| RETROACHIEVEMENTS | Syncs achievement data |
### Region Handling
```yaml
scan:
priority:
region:
- "us"
- "eu"
- "jp"
region_mode: prefer_rom_tags # or prefer_config
```
- `prefer_rom_tags` (default): ROM filename region tags take precedence.
- `prefer_config`: Config regions come first.
- **NTSC-U** (USA): 60 Hz, typically fastest, most translations.
- **PAL** (Europe): 50 Hz, sometimes slower, may have bonus content.
- **NTSC-J** (Japan): 60 Hz, Japanese only unless region-patched.
## 7. Collection Management Tools
| Tool | Primary Role | Platform |
|------|-------------|----------|
| RomVault 3.7.6 | DAT-file-based verification and reorganization | Windows |
| clrMAMEPro | ROM set validation, deduplication, 1G1R | Windows |
| DATBios | BIOS file organization using DAT files | Windows |
| Batocera/Lakka integration | ES-DE gamelist import from RomM | Linux/ARM |
## 8. Environment Variables (Dankserver)
```bash
# Auth
ROMM_USERNAME=admin
ROMM_PASSWORD=<strong password>
ROMM_AUTH_SECRET_KEY=$(openssl rand -hex 32)
# Metadata Providers
IGDB_CLIENT_ID=<your-client-id>
IGDB_CLIENT_SECRET=<your-client-secret>
SCREENSCRAPER_USER=<screenscraper-username>
SCREENSCRAPER_PASSWORD=<screenscraper-password>
MOBYGAMES_API_KEY=<mobygames-key>
RETROACHIEVEMENTS_API_KEY=<retroachievements-api-key>
STEAMGRIDDB_API_KEY=<steamgriddb-key>
# Feature Flags
HASHEOUS_API_ENABLED=true
PLAYMATCH_API_ENABLED=true
# Scanning
ENABLE_RESCAN_ON_FILESYSTEM_CHANGE=true
ENABLE_SCHEDULED_RESCAN=true
REFRESH_RETROACHIEVEMENTS_CACHE_DAYS=7
```
## See Also
- [[RomM]] — main service documentation
- [[Dankserver Stack]]
- [[ARR-Stack]] — Readarr for ebook auto-downloads alongside ROM management
Reference in New Issue View Git Blame Copy Permalink