This is my replication of the Hyper-Connections Model using the ImageNet Dataset
Due to GPU Size constraints I chose to implement the Image Classification experiment using ViT's described by the authors in Appendix E2.
This project was created using uv and is highly recommended
After installing uv this project should run out of the box
First, you should download the ImageNet Dataset from Kaggle here
Once this is downloaded you will need to run the create_dataset_csv() script to create our datasets
You will need to set the data variable to your ImageNet location
uv run create_dataset_csv.py train
uv run create_dataset_csv.py val
Before kicking off training you should update the weights and biases variables entity and project in init_logging() in train.py to match your account.
If not using Weights and Biases (not recommended) you can set logs to False in main.py
To kick off training you can run
uv run main.py
Hyper-Connections Paper Link
Vision Transformers (ViT) Link "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale"
lucidrains Vision Transformer Implementation Github Link
@article{zhu2024hyper,
title={Hyper-connections},
author={Zhu, Defa and Huang, Hongzhi and Huang, Zihao and Zeng, Yutao and Mao, Yunyao and Wu, Banggu and Min, Qiyang and Zhou, Xun},
journal={arXiv preprint arXiv:2409.19606},
year={2024}
}@article{dosovitskiy2020image,
title={An image is worth 16x16 words: Transformers for image recognition at scale},
author={Dosovitskiy, Alexey and Beyer, Lucas and Kolesnikov, Alexander and Weissenborn, Dirk and Zhai, Xiaohua and Unterthiner, Thomas and Dehghani, Mostafa and Minderer, Matthias and Heigold, Georg and Gelly, Sylvain and others},
journal={arXiv preprint arXiv:2010.11929},
year={2020}
}
@misc{Wang2026lucidrains,
author = {Wang, Phil and Ankner, Zack and Kwon, Yonghye and {murufeng} and Moryossef, Amit and (), Minh-Long Luu and Sastry, Srikumar and Walton, Steven and {roydenwa} and Hassani, Ali and Lukin, Artem and sameeh, Baraa and Mazuz, Eyal and KP, Harikrishna and {JacobLinCool} and Chou, Jason and Tow, Jonathan and Kundert, Kale and Truong, Loc and Russell, Ryan and MAL, SOUMYADIP and {umbertov} and Tu, Zhengzhong and Mishra, Aditya and {Andrés} and {shabie}},
year = {2026},
month = {may 1},
title = {lucidrains/vit-pytorch},
url = {https://github.com/lucidrains/vit-pytorch},
howpublished = {https://github.com/lucidrains/vit-pytorch},
}