Extend: MDL clustering stop, CKY chart parser, subword RePair exposure, more evals#2
Conversation
…re, more evals + docs Per approved plan (after 'do all', full ~4.16M scale run, and cleanup PR#1): - MDL stopping in induce_classes(..., mdl=True, mdl_cost_weight=...): cheap online delta (class cost + fit via cosine proxy); merges only when shortens approx DL. Default=False preserves old. Wired in tinystories. - ConstructionGrammar(..., use_chart=True) + .parse_bits: simple pure-stdlib CKY over class-bigram rules for min-cost MDL parse (fallback to flat bigram). bits() uses it when flag set. Toy + ts scoring updated (chart often gives equal/better margins). - Subword: repair now captures 'repaired' seq (NTs); exposed in tinystories with note. Light tuple handling ready (hashable keys work in Counters/vecs/labels). - More evals: minimal_pairs generalized (sents= for mining real text); grammar_ppl helper; ts [5] now reports LM/CG/chart accuracies + heldout grammar ppl; auto_pairs demo. - Docs: README scaling/limitations updated with new flags + usage. ts banner dynamic + ext note. Comments. - Tests: CI smoke + full toy + ts subsets pass (with/without flags); extensions improve some margins (e.g. chart/CG fix the dog-run case on small data). - No breakage to defaults or stdlib/pure py contract. Larger scale already validated separately. See plan.md for details/anchors/tradeoffs. Continues the 'learning == compression' baseline toward the README next rungs (CKY, MDL, subword).
|
Warning Review limit reached
More reviews will be available in 47 minutes and 59 seconds. Learn how PR review limits work. Your organization has run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (3)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: aa1707480f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| for name, scorer in [("statistical back-off LM ", lm_score), | ||
| ("construction grammar ", cg.bits)]: | ||
| ("construction grammar ", cg.bits), | ||
| ("construction grammar+chart", (lambda s: cg.parse_bits(s)) if cg.use_chart else cg.bits)]: |
There was a problem hiding this comment.
In the self-contained solon.py demo, this row never exercises parse_bits: cg was constructed with the default use_chart=False, so the conditional selects cg.bits and the reported construction grammar+chart accuracy is just a duplicate of the flat construction-grammar scorer. This makes the chart evaluation in the demo misleading; construct/use a ConstructionGrammar(..., use_chart=True) instance for this row.
Useful? React with 👍 / 👎.
| score_cg = cg.bits | ||
| score_chart = cg_chart.bits # will use parse_bits internally | ||
| # demo generalized minimal_pairs on real heldout sents (more evals) | ||
| auto_pairs = minimal_pairs(random.Random(99), n=8, sents=test_sents[:300]) |
There was a problem hiding this comment.
auto_pairs is generated from held-out TinyStories sentences but then discarded, so the advertised generalized minimal-pair evaluation never runs; this section still reports accuracy only on the five hand-built examples. For experiment runs that rely on the new “more evals” output, the reported accuracy omits the held-out corruption test entirely.
Useful? React with 👍 / 👎.
1 participant
Implements key items from the approved plan (post 'do all' + full ~4.16M scale run + cleanup PR #1).
See plan.md (in session) and code comments for design/anchors/tradeoffs. Larger scale already run/validated separately (4.16M tokens).
Ready for review / further (variable slots, BabyLM, etc.).