DEV Community

Alex Chen
Alex Chen

Posted on

Use ChatGPT’s 5,000-Character Instructions Limit to Learn Configuration Testing

Primary source: https://openai.com/products/release-notes/.

OpenAI increased saved custom instructions to 5,000 characters for several ChatGPT plans on July 15. More space can hold better context—or contradictory rules that nobody tests.

Create instructions.txt and a dependency-free linter:

from pathlib import Path
text=Path('instructions.txt').read_text()
rules=[x.strip() for x in text.splitlines() if x.strip().startswith('- ')]
assert len(text)<=5000
assert len(rules)==len(set(rules)), 'duplicate rule'
for a,b in [('always include links','never include links')]:
    assert not (a in text.lower() and b in text.lower()), f'conflict: {a} / {b}'
print({'characters':len(text),'rules':len(rules)})
Enter fullscreen mode Exit fullscreen mode

Add fixtures for 5,001 characters, duplicated rules, an explicit conflict, and Unicode text. Then define three prompts with machine-checkable outputs and run them after every instruction edit.

Check Why
size catches UI rejection
conflicts prevents impossible policy
examples tests behavior, not prose
version makes changes reviewable

This linter cannot predict model compliance and the release note does not promise deterministic instruction following. What rule in your current configuration has no corresponding test?

Top comments (0)