Hi everyone in the PyTorch community!
I’m excited to share an open-source library I’ve been building called dataplan.
When building and fine-tuning models, the biggest bottleneck isn’t usually the training loop or compute—it’s data. Existing synthetic data scripts often suffer from API rate limits dropping dataset sizes below target, mode collapse, or security risks when handling sensitive code and domains.
I built dataplan to solve this by turning synthetic data generation into a deterministic, self-healing pipeline with strict mathematical guarantees.
Key Features
-
Exact Row-Count Precision (
.size()): Uses adaptive deficit loops and dynamic batching. If an API call fails or returns malformed JSON, the engine automatically calculates the shortfall and refills the queue until your exact target row count is locked in. -
Shannon Entropy Filtering: Automatically evaluates information density to mathematically drop low-signal, repetitive, or lazy model outputs.
-
Dual-Model Collaboration: Synchronizes multiple models as a “Generator-Auditor” team to handle rate-limiting, quality auditing, and fault tolerance on the fly.
-
Zero Data Leakage (
.air_gapped()): Fully supports local-first inference engines (like vLLM, Ollama, and Llama.cpp) so you can generate datasets entirely offline with zero external network calls.
Example Usage
Python
from dataplan import DataPlan
dataset = (
DataPlan()
.idea("Custom PyTorch custom autograd and tensor optimization scenarios")
.air_gapped(True)
.use_local(engine="vllm", model="llama3")
.size(5000)
.create()
)
Feedback & Contributions Welcome
dataplan is built to bridge the gap between human ideas and production-ready training data for the open-source ecosystem.
I would love for you to check it out, test it with your local or cloud pipelines, and share your feedback. Contributions, feature ideas, and PRs are extremely welcome! If you are experienced with async Python, distributed pipelines, or LLM inference and want to help shape the project as a co-maintainer, feel free to reach out or open an issue on GitHub.
Looking forward to hearing your thoughts!