Developing India's First National AI Data Repository: AIKosh

AIKosh serves as a single destination for datasets, AI models, toolkits, and use cases, eliminating the silos that previously existed across ministries and organizations. By offering a central hub, it allows researchers, startups, and enterprises to discover and leverage resources from multiple domains without duplication of effort. The repository is searchable by sector, organization, and usage tags, ensuring ease of discovery and efficient knowledge sharing.

The platform introduced structured roles – Explorer, Contributor, Organization Admin, and Platform Admin—each with distinct privileges. Contributors could upload datasets, models, and use cases, while organization admins oversaw approvals and access requests. This model ensured that data sharing remained secure and accountable, while still giving contributors flexibility to decide whether their artefacts were open, request-based, or private.

Every dataset or model published on AIKosh passed through a rigorous multi-step review process. Uploads were first validated by organization admins, followed by platform moderators, and finally the Ministry, before being visible on the platform. This ensured authenticity, compliance, and consistency across all artefacts, building trust for users who rely on verified resources.

AIKosh supported multiple methods of onboarding datasets, manual uploads, push and pull APIs, secure SFTP transfers, and direct integrations with popular platforms like GitHub, Kaggle, and Hugging Face. This flexibility reduced barriers for contributors, enabling both government ministries and private organizations to seamlessly bring their data into the ecosystem. API keys further allowed programmatic access, enabling developers to integrate datasets directly into their workflows.

To encourage reliability, Daffodil implemented automated pipelines that scored datasets on factors such as uniqueness, completeness, frequency of updates, and absence of duplication. Contributors received a star-rating, motivating them to improve data quality over time. 

Each dataset was published under a clearly defined license, ranging from open-source to restricted or government-specific terms. This provided transparency on how data could be accessed, shared, or reused. Anonymization tools were also included to safeguard sensitive information, ensuring compliance with national data protection policies and ethical AI practices.

One of AIKosh’s key impactful features was its GPU-enabled notebook, which gave users a ready-to-use sandbox for experimentation. Instead of outsourcing expensive GPUs, researchers and engineers could directly train and test AI models on the platform. This lowered costs and accelerated the development cycle, making AI research more accessible to a wider community.