Foundational models

Popularized through LLMs, GPT-3 paper (Brown et al., 2020)

Though, it should be thought as Intelligence amplification rather than “artificial intelligence” system.

Scaling laws

Initial Scaling Laws for Neural Language Models arXiv (Kaplan et al., 2020) from OpenAI

Distributed serving of large models requires cost-efficient methods¹

Petals: a decentralized system that run Llama 2 over internet

large world models

LWM: implementation of RingAttention

visions

Bibliographie

Borzunov, A., Ryabinin, M., Chumachenko, A., Baranchuk, D., Dettmers, T., Belkada, Y., Samygin, P., & Raffel, C. (2023). Distributed Inference and Fine-tuning of Large Language Models Over The Internet. arXiv preprint arXiv:2312.08361 [arXiv]
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., … Amodei, D. (2020). Language Models are Few-Shot Learners. arXiv preprint arXiv:2005.14165 [arXiv]
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020). Scaling Laws for Neural Language Models. arXiv preprint arXiv:2001.08361 [arXiv]

Distributed Inference and Fine-tuning of Large Language Models Over The Internet (Borzunov et al., 2023) ↩