Files
serverAppConfigs/serverAppConfigs-docs/Dankserver/RomM-Scraping.md
T
Frank a5fe045b58 docs(vault): one fact, one owner — repair the doubled pages, single-source the facts
Structural repair:
- 13 pages were the same document twice (each exactly 2x its HEAD size),
  a second frontmatter block whose closing fence had been glued onto a body
  line by a frontmatter rewrite that re-read its own output. Un-doubled by
  cutting at the second frontmatter and keeping the copy whose body is a
  superset; the orphan census reports 0 lost tokens against HEAD.
- log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first —
  66 dated headings merged into 34, titles demoted to bold leads, every
  content line verified present. The file had been descending at the head
  and ascending at the tail.

Ownership moves:
- New `LLM/frank-medium.md` is the locator for the retired Dankserver
  endpoint: era windows quoted from each era page's own banner, no numbers
  restated, and the two unresolved date conflicts recorded instead of
  silently resolved. `frank-archon.md` links to it instead of restating the
  pre-rename window.
- The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone;
  `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor
  verdicts and cross-split spec decode, and the four links that pointed at
  the stubs were repointed at it or at the drafter page.
- Authentik's three stub pages fold into `Authentik.md`, with the process
  detail and the outpost reference split out so the parent stays under the
  length hint.

Verified corrections (host census + live probe, 2026-10-19):
- The MXFP4 checkpoint two pages called the "on-disk rollback" is gone:
  /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on
  the era page that owns that payload; the swap page now links to it.
- frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2
  arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run
  census is kept once (155 requests) instead of twice (~120 and 155).
- Fleet shapes re-confirmed against the live endpoints before touching any
  page: 3x200000 fp8 KV, 2x200704, 2x262144.

Index rows are regenerated, not hand-carved. Net -1,849 vault lines.
2026-10-19 05:00:43 -07:00

11 KiB

type, description, tags, timestamp
type description tags timestamp
reference Metadata providers, cover art, ROM naming and the scraping workflow behind the RomM library.
dankserver
romm
2026-08-28

RomM Scraping & Asset Management

This guide covers metadata providers, cover art, ROM naming conventions, emulator integration, and the scraping workflow for RomM on Dankserver.

Reference: https://docs.romm.app

1. Metadata Providers

RomM v5.1.0+ supports 11+ metadata providers. The scan.priority.metadata list determines which provider wins for descriptive fields. The scan.priority.artwork list controls cover art sourcing separately.

Screenscraper (ScreenScraper.fr)

  • Type: Free, community-driven (French provider)
  • Credentials: SCREENSCRAPER_USER + SCREENSCRAPER_PASSWORD (account user/pass)
  • Registration: https://www.screenscraper.fr/membreinscription.php
  • Rate limits: No documented hard cap. Community-claimed ~5 req/sec burst.
  • Capabilities: Titles, descriptions, cover art, screenshots, manuals, 3D boxes, CD/cartridge covers, marquee logos, fanart, bezels, gameplay videos.
  • Region support: Excellent. Respects scan.priority.region and scan.priority.region_mode.
  • Best for: Richest media coverage (all art types), multi-platform, free. Use as secondary to IGDB for metadata, or swap priority order if Screenscraper art quality is preferred.

IGDB (Internet Game Database / Twitch)

  • Type: Freemium (free tier with limits; paid for higher volume)
  • Credentials: IGDB_CLIENT_ID + IGDB_CLIENT_SECRET (Twitch OAuth2)
  • Registration: Twitch Developer Portal. Create app with name romm-<random hash>, OAuth redirect localhost, category Application Integration, client type Confidential.
  • Rate limits: ~10 req/sec for free tier. Cached responses reduce effective calls.
  • Capabilities: Titles, descriptions, cover art, screenshots, related games, release dates, genres, platforms.
  • Limitations: No 3D boxes or physical media art. No manuals. No marquee/fanart.
  • Default priority: Highest (igdb listed first in both metadata and artwork priority).

MobyGames

  • Type: Paid (hobbyist API subscription; non-commercial use only)
  • Credentials: MOBYGAMES_API_KEY
  • Rate limits: Hobbyist tier ~0.2 req/sec (one request every 5 seconds).
  • RomM recommendation: "We recommend using ScreenScraper instead, as it is free to use."
  • Best for: Supplemental metadata depth for older PC/console titles.

RetroAchievements (RAHasher)

  • Type: Free, hash-based achievement tracking
  • Credentials: RETROACHIEVEMENTS_API_KEY
  • How it works: RomM bundles RAHasher (v1.8.3+). ROMs are hashed during scans, matched against RA online database. Achievement progress synced per-user via their RA username.
  • Known issues: Folder-based disc ROMs (.cue + .bin, .gdi) may fail hashing. Non-zip archives (.7z, .rar) had incorrect hash computation. Ensure filesystem.skip_hash_calculation is false.
  • Important: Achievements cannot be earned via RomM's built-in EmulatorJS player. Must use an RA-capable standalone emulator.

Additional Built-in Providers

Provider Slug Enabled Via Type
Hasheous hasheous HASHEOUS_API_ENABLED=true Free, hash-based, proxies IGDB data
Playmatch playmatch PLAYMATCH_API_ENABLED=true Multi-provider hash matching
LaunchBox launchbox LAUNCHBOX_API_ENABLED=true + XML Local XML database
Flashpoint flashpoint FLASHPOINT_API_ENABLED=true 180,000+ browser games
HowLongToBeat hltb HLTB_API_ENABLED=true Game completion times
SteamGridDB steamgriddb STEAMGRIDDB_API_KEY Custom cover art
ES-DE gamelist gamelist Place gamelist.xml in platform folder Local XML
Combo Providers Best For Cost
Quick Starter Hasheous only No API keys needed Free
French Connection Screenscraper + RetroAchievements Rich media, free Free
Chef's Choice (recommended) Hasheous + IGDB + SteamGridDB + RetroAchievements Best overall quality Free (IGDB free tier)
Twitch Fanboy IGDB + Playmatch Single-provider solution Free

For Dankserver, the recommended setup:

  • IGDB (primary metadata) + SCREENSCRAPER (artwork backup + rich media) + RETROACHIEVEMENTS (achievements) + HASHEOUS_API_ENABLED=true (fast hash-based fallback)

3. ROM File Naming Conventions

The de facto standard for cartridge and handheld ROMs. Only Title and Region are required.

Super Metroid (Japan, USA) (En,Ja)
Sonic the Hedgehog (USA)
Metal Gear Solid (USA) (Disc 1)
[BIOS] PS1 System BIOS (USA)
Tetris [!].gba

Field meanings:

  • (Region): USA, EUR, JPN, UE (USA+Europe), JUE (Japan+USA+Europe)
  • (Languages): En, Ja, De, Fr, Es
  • [Status]: [!] = verified good dump, [b] = bad dump, [c] = completed
  • (Devstatus): (Proto), (Demo), (Release Candidate)
  • [BIOS] prefix: Identifies BIOS files

RomM's Region/Language/Revision Parsing

RomM parses inline tags from filenames:

tetris (En) [1.0001] [HACK] [!].gba
  • Region/language: ISO-like codes or full names
  • Revision: rev v1 or rev-1
  • Provider ID tags: (igdb-xxxx), (ssfr-xxxx), (moby-xxxx), (ra-xxxx) — forces specific provider match

Best Practices for Dankserver

  1. Use No-Intro naming as the standard.
  2. Use Igir to rename and organize ROMs into RomM's required folder structure:
    igir copy zip test clean \
      --dat "No-Intro*.zip" \
      --input "ROMs" \
      --output "/opt/romm/roms/{romm}"
    
  3. Keep [BIOS] prefix for BIOS files in the bios/ directory.
  4. One ROM file per game folder where possible.
  5. Use archive formats (.zip, .7z) — RomM supports them natively.

4. Cover Art & Media Assets

Media Types

Media Type Description Best Provider
box2d Standard 2D front cover art IGDB, Screenscraper
box3d 3D box art Screenscraper
miximage Composite (box + screenshot + logo) Screenscraper
screenshot In-game screenshot IGDB, Screenscraper
marquee Transparent logo/header Screenscraper
bezel EmulatorJS-compatible bezel Screenscraper
manual PDF instruction manual Screenscraper
video Gameplay video/trailer Screenscraper

Custom Artwork Directories

RomM supports manual artwork placement via subdirectories within each game folder:

/opt/romm/library/roms/{platform}/{game}/
    game.rom
    covers/
    screenshots/
    3dboxes/
    manuals/
    fanart/
    videos/
    marquees/
    bezels/

For ES-DE gamelist imports, media maps as:

gamelist.xml Tag Folder Name
<cover> covers/
<backcover> backcovers/
<box3d> 3dboxes/
<marquee> marquees/
<screenshot> screenshots/
<fanart> fanart/
<manual> manuals/

MAME Artwork

MAME uses cabinet pics, flyers, marquees, control panels, backdrops, overlays. Stored in {platform}/mame_artwork/{game}/ subdirectories.

Sources: Mr. Do's Arcade (https://www.mameworld.info/mrdo/), ProgettoSnaps (http://www.progettosnaps.net/artworks/).

5. Emulator Integration

EmulatorJS (Built-in)

Web-based player using RetroArch cores compiled to WebAssembly. Runs in-browser with no local emulator needed. Configured via emulatorjs.* settings in config.yml. Supports netplay via STUN/TURN servers.

Emulator Streaming

Launch native emulator containers (PCSX2, Dolphin, Xemu) and stream picture/sound/input back to the browser. Configured via streaming.* in config.yml.

BIOS File Requirements

BIOS files placed in /opt/romm/library/bios/{platform}/. Platform subdirectories required. File names should match emulator expectations:

Platform BIOS File
PlayStation scph1001.bin, scph5501.bin, scph5502.bin
GB/GBC gb_bios.bin, gbc_bios.bin
SNES sfc_bios.bin

BIOS matching done by hash against platform requirements.

Folder Structure Options

Structure A (recommended):

/opt/romm/library/
    roms/
        └── {platform}/
            └── {game}.rom
    bios/
        └── {platform}/
            └── {bios_file}.bin

Structure B (fallback):

/opt/romm/library/
    └── {platform}/
        ├── roms/
        │   └── {game}.rom
        └── bios/
            └── {bios_file}.bin

Dankserver mounts /opt/romm/library:/romm/library — RomM auto-detects structure.

6. Scraping Workflow

  1. Organize ROMs → No-Intro naming + RomM folder structure.
  2. Create config.yml → Set provider credentials, priority lists, region prefs.
  3. First scan → Scan ALL → verify everything is matched.
  4. Review unmatched → Manually match or add provider tags to filenames.
  5. Fetch artwork → Re-scan with scan.media extended to desired types.
  6. Sync RetroAchievements → Set user RA username → trigger sync.
  7. Export gamelist → Enable scan.gamelist.export for ES-DE/Batocera.
  8. Maintain → Enable scheduled rescans for new DAT updates.

Scan Types

Scan Type What It Does
ALL Full scan — hash, match metadata, fetch artwork
UNMATCHED Only matches unmatched ROMs (faster)
METADATA Updates metadata only
ARTWORK Re-fetches artwork only
RETROACHIEVEMENTS Syncs achievement data

Region Handling

scan:
  priority:
    region:
      - "us"
      - "eu"
      - "jp"
    region_mode: prefer_rom_tags   # or prefer_config
  • prefer_rom_tags (default): ROM filename region tags take precedence.
  • prefer_config: Config regions come first.
  • NTSC-U (USA): 60 Hz, typically fastest, most translations.
  • PAL (Europe): 50 Hz, sometimes slower, may have bonus content.
  • NTSC-J (Japan): 60 Hz, Japanese only unless region-patched.

7. Collection Management Tools

Tool Primary Role Platform
RomVault 3.7.6 DAT-file-based verification and reorganization Windows
clrMAMEPro ROM set validation, deduplication, 1G1R Windows
DATBios BIOS file organization using DAT files Windows
Batocera/Lakka integration ES-DE gamelist import from RomM Linux/ARM

8. Environment Variables (Dankserver)

# Auth
ROMM_USERNAME=admin
ROMM_PASSWORD=<strong password>
ROMM_AUTH_SECRET_KEY=$(openssl rand -hex 32)

# Metadata Providers
IGDB_CLIENT_ID=<your-client-id>
IGDB_CLIENT_SECRET=<your-client-secret>
SCREENSCRAPER_USER=<screenscraper-username>
SCREENSCRAPER_PASSWORD=<screenscraper-password>
MOBYGAMES_API_KEY=<mobygames-key>
RETROACHIEVEMENTS_API_KEY=<retroachievements-api-key>
STEAMGRIDDB_API_KEY=<steamgriddb-key>

# Feature Flags
HASHEOUS_API_ENABLED=true
PLAYMATCH_API_ENABLED=true

# Scanning
ENABLE_RESCAN_ON_FILESYSTEM_CHANGE=true
ENABLE_SCHEDULED_RESCAN=true
REFRESH_RETROACHIEVEMENTS_CACHE_DAYS=7

See Also