Overview
I am Anjani Nandan, and my Google Summer of Code (GSoC) journey has a lot of meaning for me.
Before GSoC, I had almost no knowledge about what GSoC was or how open-source development actually worked. When I started this journey, my main expectation was to learn technical skills and contribute to a real software project. However, by the end of the program, I realized that GSoC gave me much more than technical knowledge.
This report is not only about what I implemented technically. It is also about how GSoC positively impacted me as a person and as a future software engineer.
One of the biggest things I learned was how to work professionally: how to communicate with mentors, how to discuss technical decisions, how to take responsibility for my work, and how to think about a problem before jumping into implementation.
My mentors played an immense role in transforming many of my naive ideas into much more mature approaches. What I appreciated most was the freedom they gave me to think independently.
Instead of always telling me what to implement, they often gave me a problem, explained the context, and allowed me to explore possible solutions on my own. I would then come back with my understanding, ideas, and questions, and we would discuss them together.
This taught me an important lesson: independent thinking does not mean working alone. Good teamwork can actually create space for independent thinking while still providing collaboration, discussion, and feedback when needed.
During the project, I also encountered and learned several concepts and tools that were new to me. Some of the important ones were:
Symbolic links (symlinks): A symbolic link is a special type of file that points to another file or directory. I used this concept while working on the caching system, where symlinks can help avoid unnecessary duplication of dataset files while still making them accessible from different locations.
Pagination: APIs may return a large number of results, so instead of returning everything in a single response, the results are divided into smaller pages. I learned how to work with paginated Hugging Face APIs to retrieve datasets in manageable chunks.
vcr:vcris an R package used for recording HTTP requests and their responses so that they can be replayed during testing. This allows tests involving external APIs to be more reliable and avoids making unnecessary network requests every time the tests are run.Caching for dataset revisions: A dataset can have multiple versions or revisions over time. I learned how to design a caching system that can distinguish between different revisions, including specific commit hashes, so that users can reliably access the version of the dataset they requested.
pkgdown:pkgdownis an R tool for automatically generating websites and documentation for R packages. I learned how package documentation, reference pages, articles, and other resources can be organized and published as a documentation website.Parallel API fetching: When working with APIs, making requests one after another can be slow, especially when many independent requests are required. I learned how to perform multiple API requests concurrently to improve the efficiency of operations such as fetching dataset information.
Other packages in the
mlr3ecosystem: I also explored related packages such asmlr3andmlr3learnersto understand how the different components of the ecosystem work together. This helped me understand how datasets are represented as Tasks and how those Tasks can subsequently be used with machine learning learners and othermlr3functionality.
Working on a real project helped me understand not only what these concepts mean, but also why and when they are useful in real software systems.
I worked on the repository mlr3hf.
You can also visit the package documentation. A development version of the package has been released on r-universe.
We are currently planning to submit the package to CRAN after making some final refinements.
What Is the Project?
Before introducing my project, I would like to briefly explain what Hugging Face is and why datasets from Hugging Face are useful.
Hugging Face is a platform that provides a large collection of datasets, models, and other resources that are widely used in machine learning and deep learning. In my project, I mainly focused on the Hugging Face Hub for datasets. You can visit the Hugging Face website to explore these resources.
So, why do we need these datasets?
Researchers, students, and developers often need datasets to experiment with, develop, and evaluate machine learning models. However, simply downloading a dataset is not always enough. In the R ecosystem, users of mlr3 work with datasets through a specific abstraction called a Task.
An mlr3 Task contains the dataset together with information about its features, target variable, and other relevant metadata. This allows the dataset to be directly used with the different components of the mlr3 ecosystem.
This is where mlr3hf comes in.
The goal of my project was to develop an R package that allows users to access datasets from the Hugging Face Hub and convert them into objects that can be used within the mlr3 ecosystem.
My Contributions
Since the project was developed largely from scratch, I worked on several different parts of the package, including dataset downloading, caching, metadata handling, Parquet access, testing, and conversion into mlr3 objects.
During GSoC, I opened more than 20 pull requests and issues, covering different aspects of the implementation, testing, documentation, CI, and project infrastructure.
A complete list of my pull requests and issues is provided below.
Pull Requests and Issues
I opened more than 20 pull requests and issues throughout the GSoC period.
Every pull request that I submitted was eventually merged, and there are currently no pending changes from my GSoC work upstream.
| PR/Issue | Description | Status | Link |
|---|---|---|---|
| PR #3 | Implementing dataset downloads and fixing R CMD check issues |
Merged | PR #3 |
| PR #4 | Adding a dependency for dataset descriptions | Merged | PR #4 |
| PR #6 | Adding pkgdown |
Merged | PR #6 |
| PR #9 | Implementing dataset caching | Merged | PR #9 |
| PR #10 | Adding Netlify integration | Merged | PR #10 |
| PR #12 | Adding GitHub Pages and preview comments to show changes | Merged | PR #12 |
| PR #15 | Implementing list_datasets() |
Merged | PR #15 |
| PR #16 | Removing the built-in container from the workflows | Merged | PR #16 |
| PR #20 | Fixing skip() and adding vcr |
Merged | PR #20 |
| PR #22 | Adding the caching design documentation | Merged | PR #22 |
| PR #25 | Resolving Issue #24 | Merged | PR #25 |
| PR #26 | Adding backend_parquet.R and backend_hfhub.R |
Merged | PR #26 |
| PR #27 | Task creation and vignettes | Merged | PR #27 |
| PR #29 | fixing file_name in url | Merged | PR #29 |
| PR #30 | solving issues 17 | Merged | PR #30 |
| PR #33 | correcting subdirectory file_name | Merged | PR #33 |
Challenges I Faced
The biggest challenge for me was not writing individual functions. It was making design decisions.
When developing a package from scratch, there are often several ways to solve the same problem. Choosing one approach requires thinking about future users, maintainability, performance, edge cases, and how the different components of the package will interact.
The caching design was a particularly challenging part.
I also had to learn how real open-source projects work:
- understanding an existing codebase,
- reading documentation and source code,
- discussing design decisions,
- writing tests,
- responding to code review,
- debugging unexpected behavior,
- and thinking about how future contributors will understand the code.
At several points, I was unsure whether my approach was the right one. Instead of simply giving me the answer, my mentors encouraged me to think through the problem and explain my reasoning.
That process helped me become more comfortable with uncertainty.
I learned that not knowing the answer immediately is normal in software engineering. What matters is developing the ability to investigate, reason, experiment, discuss, and eventually arrive at a good solution.
What I Learned Beyond Technical Skills
GSoC taught me several things that I don’t think I could have learned as effectively from a course or textbook.
1. How to Communicate With Mentors
Before GSoC, I did not have much experience discussing technical problems with experienced software developers.
I learned how to explain:
- what I understood,
- what I had tried,
- where I was stuck,
- why I preferred a particular approach,
- and what trade-offs I was considering.
This made my technical communication much clearer.
2. Think Independently
My mentors gave me significant freedom to explore solutions before discussing them.
This taught me to first ask myself:
“What do I think the solution should be, and why?”
Only after developing my own understanding would I discuss it with my mentors.
This habit has become one of the most valuable outcomes of GSoC for me.
Current State of the Package
At the end of GSoC, the major planned components of the project have been implemented.
The package currently provides functionality for downloading and accessing datasets from Hugging Face, caching them locally, working with different revisions, and converting the data into objects usable by the mlr3 ecosystem.
However, I do not consider the project “finished,” in the sense that no software project is ever completely finished.
A few concrete areas remain open for future work:
- support for additional dataset formats beyond Parquet, csv and tsv,
- broader test coverage for edge cases in the caching layer,
- and continued documentation updates as the mlr3 ecosystem evolves.
If you find a bug, or see an area that can be improved, raise an issue, contributions are welcome.
Acknowledgements
I sincerely acknowledge my mentors, Toby Dylan Hocking and Sebastian Fischer, for their guidance and support throughout this journey. I am also grateful to The R Project for Statistical Computing for providing me with the opportunity to be part of this experience.
However, I want to acknowledge my mentors for something beyond simply helping me complete the project.I would also like to acknowledge the Hugging Face documentation, and the other dependencies used by this package, which were valuable resources throughout the development of mlr3hf.