Use when adding new AI/startup directories to directories.json from a URL, pasted text, or GitHub awesome-list. Parses, deduplicates, appends new entries, th...
directories.jsonFetch the page and extract directory entries. Look for patterns like:
Use gh CLI to explore curated lists:
gh repo clone <owner>/<repo> to clone awesome-listsParse lines/rows. Common formats:
Name - https://url.com or Name | https://url.com[Name](https://url.com)Extract at minimum: name and url (submission or homepage).
Load directories.json and check each parsed entry against existing ones by:
Report duplicates to the user and skip them.
For each new directory, create an entry with this structure:
{
"categories": ["General"],
"description": "",
"is_active": true,
"name": "Directory Name",
"pricing_type": "free",
"slug": "directory-name",
"submission_url": "https://example.com/submit",
"url": "https://example.com/submit"
}
Field rules:
slug: lowercase name, spaces to hyphens, strip special charssubmission_url and url: use the submission/signup URL if available, otherwise homepagedescription: leave empty string (will be filled later or by user)categories: default ["General"] unless context provides a categorypricing_type: default "free" unless explicitly marked paidis_active: always true for new entriesSave the updated directories.json.
Run the pipeline scripts in order using the project venv at .venv/:
# 1. HTTP-level analysis (auth, captcha, pricing signals, dead domains)
.venv/bin/python analyze_directories.py
# 2. Cleanup obvious failures + build browser check list
.venv/bin/python cleanup_and_categorize.py
# 3. Browser verification with Playwright (10 concurrent workers)
.venv/bin/python browser_verify.py
# 4. Deep recheck any remaining unknowns
.venv/bin/python browser_verify.py --recheck-unknown
Each script reads/writes directories.json. Steps 3-4 use browser_check_list.json as intermediate state (generated by step 2).
After completion, report the summary: how many added, and the auth/status breakdown for the new entries.
For directories that are active and have auth_type = none or auth_type = email_password:
# Discover form fields on submission pages
.venv/bin/python discover_forms.py
This visits each submission URL with Playwright, extracts form fields via DOM queries, and updates submission_plan.json with discovered fields and form paths.
Configure the PRODUCT dict in submit_directories.py with your details (search for YOUR_ placeholders), then:
# Auto-submit to all discovered directories
.venv/bin/python submit_directories.py
The script uses heuristic field mapping (matching field names/labels to product data) and handles file uploads for logo/screenshot.
For directories that need manual interaction (captcha, OAuth, complex forms), use the Playwright browser tools:
browser_fill_form or browser_typebrowser_file_uploadSome directories accept submissions via GitHub PRs to awesome-lists:
gh repo fork <owner>/<repo>gh pr createanalyze_directories.py uses ThreadPoolExecutor with plain HTTP — fast first passcleanup_and_categorize.py triages errors (dead domains, invalid URLs, Facebook groups) and builds browser_check_list.jsonbrowser_verify.py uses async Playwright with 10 concurrent tabs; --recheck-unknown does a deep DOM pass on active unknowns onlydiscover_forms.py uses async Playwright with 10 concurrent tabs; extracts form field names, types, labels, and pathssubmit_directories.py uses async Playwright with 5 concurrent tabs; heuristic field mapping with file upload supportWhen evaluating or submitting to directories, watch for these issues:
| Blocker | Frequency | How to Detect |
|---|---|---|
| Paid listing required | ~20% | Look for pricing page, Stripe/PayPal links, "$" on submit page |
| reCAPTCHA / Turnstile | ~10% | iframe[src*=recaptcha] or [data-turnstile] elements |
| Broken captcha | ~2% | "Invalid site key" errors, disabled submit buttons |
| Login/account required | ~15% | Redirect to /login or /register on submit URL |
| Business email required | ~3% | Rejects gmail/yahoo domains (e.g., SoftwareSuggest) |
| Reciprocal link required | ~5% | Old web directories require backlink before listing |
| Newsletter-only forms | ~10% | Page looks like submit but is actually email signup |
| Backend API broken | ~2% | Form submits but returns GraphQL/API errors |
| Domain parked/dead | ~8% | No content, parking page, DNS failure |
| Cloudflare blocked | ~3% | Challenge page, 403 errors |
browser_evaluate to set contentbrowser_tabs action to handle Google loginbrowser_click on the dropdown, then click the optionEach entry in submission_plan.json contains:
{
"directory_name": "Example AI",
"submission_url": "https://example.com/submit",
"status": "discovered",
"copy": {
"title": "Product Title Variation",
"description": "Product description variation for this directory."
},
"discovered_fields": [...],
"form_path": "form#submit-form",
"credentials": {
"email": "YOUR_EMAIL",
"name": "YOUR_NAME",
"username": "YOUR_USERNAME",
"password": "YOUR_PASSWORD"
}
}
Status values: discovered, submitted, skipped, skipped_paid, timeout, no_form_found, no_fields_matched, submit_timeout, captcha, cloudflare_blocked, domain_parked, skipped_login_required, deferred.
Before pushing to GitHub, ensure all personal data is stripped:
YOUR_ placeholders in submission_plan.json and submit_directories.py.playwright-mcp/ folder may contain console logs with personal data — add to .gitignoreZIP package — ready to use