A Scalable Cloud-based Architecture to Deploy JupyterHub for Computational Social Science Research

被引:0
作者
Li, Da [1 ]
Pyke, Robert [1 ]
Jiang, Runchao [1 ]
机构
[1] Facebook Inc, Menlo Pk, CA 94025 USA
来源
PRACTICE AND EXPERIENCE IN ADVANCED RESEARCH COMPUTING 2021, PEARC 2021 | 2021年
关键词
Jupyter; Kubernetes; Cloud Computing; Scalability; Computational Social Science;
D O I
10.1145/3437359.3465591
中图分类号
TP31 [计算机软件];
学科分类号
081202 ; 0835 ;
摘要
With the increasing popularity of computational approaches to conduct social science research, building a scalable and efficient computing platform has become a topic of interest for academia to empower research labs and institutes to analyze large-scale data. While social science researchers have been very excited about the advancement of emerging technologies in big data, deep learning, computer vision, network analysis, etc., they are also constrained by the available computing resources to analyze data. This paper describes a scalable solution to deploy JupyterHub for computational social science research on the cloud. We use a reference architecture on AWS to walk through the design principles and details. Our architecture has helped facilitate several collaborations between Facebook and academia. The case study (Facebook Open Research and Transparency platform) shows that our architecture, using technologies like containerization and serverless computing, can support thousands of users to analyze web-scale datasets.
引用
收藏
页数:4
相关论文
共 10 条
  • [1] [Anonymous], 2020, NCI Cloud Resources
  • [2] [Anonymous], 2020, New privacyprotected Facebook data for independent research on social media's impact on democracy
  • [3] [Anonymous], 2021, OpenAPI Initiative
  • [4] [Anonymous], 2017, Zero to JupyterHub with Kubernetes
  • [5] [Anonymous], 2020, SVInfer
  • [6] [Anonymous], 2021, Introducing new electionrelated ad data sets for researchers
  • [7] Jupyter Kernels, 2015, Wiki: Jupyter Kernels
  • [8] JupyterHub, 2021, JupyterHub: A multiuser version of the notebook designed for companies, classrooms and research labs
  • [9] Jupyter Notebooks-a publishing format for reproducible computational workflows
    Kluyver, Thomas
    Ragan-Kelley, Benjamin
    Perez, Fernando
    Granger, Brian
    Bussonnier, Matthias
    Frederic, Jonathan
    Kelley, Kyle
    Hamrick, Jessica
    Grout, Jason
    Corlay, Sylvain
    Ivanov, Paul
    Avila, Damin
    Abdalla, Safia
    Willing, Carol
    [J]. POSITIONING AND POWER IN ACADEMIC PUBLISHING: PLAYERS, AGENTS AND AGENDAS, 2016, : 87 - 90
  • [10] Prime ministers, the vote of confidence and the management of coalition terminations between elections
    Schleiter, Petra
    Evans, Georgina
    [J]. WEST EUROPEAN POLITICS, 2022, 45 (03) : 528 - 549