tl;dr

Last weekend, we were doing a team hackathon, with everyone just trying to tinker with ideas and build something “fun”.

I (Sid) had spent a weekend in February on LLMs (specifically OpenAI) and decided to pick up the same again to explore its capabilities of understanding datasets and getting insights from them. Compared to last time (which seems like ages away in the LLM world), it’s much more logical and supports structured responses natively.

The blog is a note on how we built it.

Hackathon output:

Built a trivia game on IPL 2023 using a public dataset, OpenAI APIs and some bit of prompt engineering.

Github repo — backend, frontend. (Not adding the game link because it will otherwise consume all my OpenAI credits)

Pre-cursor:

Learnings from trying out GPT the previous time:

(a) Unstructured Responses: Hard to get structured data, which meant using it in code was not feasible (yet)

(b) Frequent Hallucination: Illogical responses / irrelevant to the question are common

(c) Expensive: to run after a point

Learnings from building the game:

The responses have been way better this time around compared to last time:

• Structured response: OpenAI itself has natively enabled functional responses, so it’s easier to now get structured responses directly.

• Logic: With GPT4, the quality of responses has seen an uptick; it can transform text to pandas queries seamlessly.

• Cost: Remains expensive as we needed to train it on the schema every time.

Tools we used:

• OpenAI Function Calling

• Kaggle Dataset — IPL 2023

• Code Stack: ReactJS, Python

• Infra Stack: Docker Compose + AWS EC2 + gunicorn (nightmare to deploy full-stack application as a side project)

Detailed process:

Step 1 — Create prompt: Dataset Schema and also provide it with some unique values for each dimension of the schema.

Our prompt:

messages = [{"role": "user",
                 "content": 
"these are 4 datasets that I have related to IPL 2023. "
"I want to ask fun trivia questions to a user basis this dataset - try a random question every time. "
"\n\nGenerate the question in function format as requested\n\n\\n\\n"


"IPL2023_Batter.csv named as df_batter\\nmatch_no - No of match according to IPL schedule\\nBatsman - Name of the Batsman in that particular match\\nteam - Batsman played in which team\\nRun - Run scored by that Batsman\\nBall- The number of balls he took to score the runs\\n4s - How much 4 he scored in that match\\n6s - How much 6 he scored in that match\\nout_by - name of bowler who outed him\\n\\n\\nIPL2023_Bowler.csv named as df_bowler\\nmatch_no - No of match according to IPL schedule\\nBowler - Name of the Bowler in that particular match\\nteam - Bowler played in which team\\nover_ - How much over he did in that match\\nrun- How much runs he gave in that match\\nwicket - How much wicket taken by that bowler\\nNo_ball - How much No ball he done\\nECO - economy rate is the average number of runs\\n\\n\\nIPL2023_Match_Scoreboard.csv named as df_scoreboard\\nmatch_no - No of match according to IPL schedule\\nHome_team_run - Total Runs done by home team\\nHome_team_wickets - Home team's total wickets gone\\nHome_team_over - Total overs played by Home team\\nAway_team_run - Total Runs done by Away team\\nAway_team_wickets - Away team's total wickets gone\\nAway_team_over - Total overs played by Away team\\n\\n\\nIPL2023_Matches.csv named as df_matches\\nmatch_no - Match number according to IPL schedule\\ncity - Location of match\\ndate_of_match - Date of Match\\nvenue - Venue of the match\\nHome_team - Team playing in home ground\\nAway_team - Team not playing in home ground\\ntoss_winner - Team that won the toss\\nwinner - Team that won the match\\nman_of_the_match - Player awarded \\\"Man of the Match\\\"\\nresult - Game won by 1st batting team or 1st bowling team\\nresult_margin - \\neliminator\\numpire1\\numpire2\n\n\n"

"Here are some data "
"Batsmen:  ['Devon Conway' 'Ruturaj Gaikwad' 'Moeen Ali' 'Ben Stokes' 'Ambati Rayudu'\n 'Shivam Dube' 'Ravindra Jadeja' 'MS Dhoni' 'Mitchell Santner'\n 'Wriddhiman Saha' 'Shubman Gill' 'Sai Sudharsan' 'Hardik Pandya'\n 'Vijay Shankar' 'Rahul Tewatia' 'Rashid Khan' 'Prabhsimran Singh'\n 'Shikhar Dhawan' 'Bhanuka Rajapaksa' 'Jitesh Sharma' 'Sikandar Raza'\n 'Sam Curran' 'Shahrukh Khan' 'Mandeep Singh' 'Rahmanullah Gurbaz'\n 'Anukul Roy' 'Venkatesh Iyer' 'Nitish Rana' 'Rinku Singh' 'Andre Russell'\n 'Shardul Thakur' 'Sunil Narine' 'KL Rahul' 'Kyle Mayers' 'Deepak Hooda'\n 'Krunal Pandya' 'Marcus Stoinis' 'Nicholas Pooran' 'Ayush Badoni'\n 'Krishnappa Gowtham' 'Prithvi Shaw' 'David Warner' 'Mitchell Marsh'\n 'Sarfaraz Khan' 'Rilee Rossouw' 'Rovman Powell' 'Aman Hakim Khan'\n 'Axar Patel' 'Kuldeep Yadav' 'Chetan Sakariya' 'Mukesh Kumar'\n 'Yashasvi Jaiswal' 'Jos Buttler' 'Sanju Samson' 'Devdutt Padikkal'\n 'Riyan Parag' 'Shimron Hetmyer' 'Ravichandran Ashwin' 'Abhishek Sharma'\n 'Mayank Agarwal' 'Rahul Tripathi' 'Harry Brook' 'Washington Sundar'\n 'Glenn Phillips' 'Abdul Samad' 'Adil Rashid' 'Bhuvneshwar Kumar'\n 'Umran Malik' 'Rohit Sharma' 'Ishan Kishan' 'Cameron Green'\n 'Suryakumar Yadav' 'Tilak Varma' 'Nehal Wadhera' 'Tim David'\n 'Hrithik Shokeen' 'Arshad Khan' 'Virat Kohli' 'Faf du Plessis'\n 'Dinesh Karthik' 'Glenn Maxwell' 'Mark Wood' 'Abishek Porel'\n 'Anrich Nortje' 'David Miller' 'Dhruv Jurel' 'Jason Holder' 'Umesh Yadav'\n 'Michael Bracewell' 'Harshal Patel' 'Shahbaz Ahmed' 'Anuj Rawat'\n 'David Willey' 'Karn Sharma' 'Akash Deep' 'Anmolpreet Singh'\n 'Aiden Markram' 'Romario Shepherd' 'Manish Pandey' 'Lalit Yadav'\n 'Tristan Stubbs' 'Piyush Chawla' 'Ajinkya Rahane' 'Abhinav Manohar'\n 'N Jagadeesan' 'Matthew Short' 'Harpreet Brar' 'Rahul Chahar'\n 'Nathan Ellis' 'Mohit Rathee' 'Jaydev Unadkat' 'Ravi Bishnoi'\n 'Avesh Khan' 'Yash Dhull' 'Mustafizur Rahman' 'Adam Zampa' 'Rishi Dhawan'\n 'Heinrich Klaasen' 'Mahipal Lomror' 'Yudhvir Singh' 'Atharva Taide'\n 'Harpreet Singh Bhatia' 'Kagiso Rabada' 'Alzarri Joseph' 'Trent Boult'\n 'Suyash Prabhudessai' 'Wayne Parnell' 'Wanindu Hasaranga' 'Marco Jansen'\n 'Mayank Markande' 'Liam Livingstone' 'Arshdeep Singh' 'Jason Roy'\n 'Litton Das' 'Varun Chakaravarthy' 'Philip Salt' 'Prerak Mankad'\n 'Harpreet Singh' 'Jofra Archer' 'Vijaykumar Vyshak' 'Mohammed Siraj'\n 'Abdul Basith' 'David Wiese' 'Ripal Patel' 'Ishant Sharma'\n 'Arjun Tendulkar' 'Jason Behrendorff' 'Riley Meredith' 'Priyam Garg'\n 'Akeal Hosein' 'Josh Hazlewood' 'Amit Mishra' 'Naveen-ul-Haq'\n 'Manan Vohra' 'Karan Sharma' 'Harshit Rana' 'Vaibhav Arora'\n 'Sandeep Sharma' 'Quinton de Kock' 'Swapnil Singh' 'Kedar Jadhav'\n 'Deepak Chahar' 'Tushar Deshpande' 'Vishnu Vinod' 'Noor Ahmad'\n 'Praveen Dubey' 'Joe Root' 'KM Asif' 'Dasun Shanaka' 'Mohammed Shami'\n 'Mohit Sharma' 'Sanvir Singh' 'Fazalhaq Farooqi' 'Vivrant Sharma']\n"
"Out by:  ['Mohammed Shami' 'Alzarri Joseph' 'Rashid Khan' 'Joshua Little' 'Not Out'\n 'Rajvardhan Hangargekar' 'Tushar Deshpande' 'Ravindra Jadeja'\n 'Tim Southee' 'Varun Chakaravarthy' 'Umesh Yadav' 'Sunil Narine'\n 'Arshdeep Singh' 'Nathan Ellis' 'Sikandar Raza' 'Rahul Chahar'\n 'Sam Curran' 'Chetan Sakariya' 'Axar Patel' 'Kuldeep Yadav'\n 'Khaleel Ahmed' 'Mark Wood' 'Avesh Khan' 'Fazalhaq Farooqi' 'T Natarajan'\n 'Umran Malik' 'Trent Boult' 'Yuzvendra Chahal' 'Jason Holder'\n 'Ravichandran Ashwin' 'Akash Deep' 'Mohammed Siraj' 'Reece Topley'\n 'Michael Bracewell' 'Karn Sharma' 'Harshal Patel' 'Arshad Khan'\n 'Cameron Green' 'Ravi Bishnoi' 'Moeen Ali' 'Mitchell Santner'\n 'Anrich Nortje' 'Mitchell Marsh' 'retd hurt(Not Out)' 'Run Out'\n 'David Willey' 'Shardul Thakur' 'Suyash Sharma' 'Krunal Pandya'\n 'Yash Thakur' 'Amit Mishra' 'Adil Rashid' 'Bhuvneshwar Kumar'\n 'Mukesh Kumar' 'Rovman Powell' 'Sandeep Sharma' 'Sisanda Magala'\n 'Jason Behrendorff' 'Piyush Chawla' 'Kumar Kartikeya' 'Marco Jansen'\n 'Mayank Markande' 'Wayne Parnell' 'Mustafizur Rahman' 'Hrithik Shokeen'\n 'Riley Meredith' 'Akash Singh' 'Adam Zampa' 'Mohit Sharma'\n 'Kagiso Rabada' 'Harpreet Brar' 'Andre Russell' 'Lalit Yadav'\n 'Vijaykumar Vyshak' 'Wanindu Hasaranga' 'Yudhvir Singh'\n 'Krishnappa Gowtham' 'Duan Jansen' 'Lockie Ferguson' 'Hardik Pandya'\n 'Noor Ahmad' 'Glenn Maxwell' 'Maheesh Theekshana' 'Matheesha Pathirana'\n 'Arjun Tendulkar' 'Marcus Stoinis' 'Ishant Sharma' 'Nitish Rana'\n 'Anukul Roy' 'Naveen-ul-Haq' 'Jofra Archer' 'Liam Livingstone'\n 'Kulwant Khejroliya' 'Washington Sundar' 'Kuldip Yadav' 'Harshit Rana'\n 'Akeal Hosein' 'Abhishek Sharma' 'Josh Hazlewood' 'Rishi Dhawan'\n 'Kartik Tyagi' 'Aiden Markram' 'Vaibhav Arora' 'Deepak Chahar'\n 'Akash Madhwal' 'Tristan Stubbs' 'Mohsin Khan' 'Chris Jordan' 'KM Asif'\n 'Glenn Phillips' 'Praveen Dubey' 'Yash Dayal' 'Retd Hurt' 'Shahbaz Ahmed'\n 'Navdeep Saini' 'Mayank Dagar']\nBowlers  ['Mohammed Shami' 'Hardik Pandya' 'Joshua Little' 'Rashid Khan'\n 'Alzarri Joseph' 'Yash Dayal' 'Deepak Chahar' 'Tushar Deshpande'\n 'Rajvardhan Hangargekar' 'Mitchell Santner' 'Ravindra Jadeja'\n 'Umesh Yadav' 'Tim Southee' 'Sunil Narine' 'Varun Chakaravarthy'\n 'Shardul Thakur' 'Sam Curran' 'Arshdeep Singh' 'Nathan Ellis'\n 'Sikandar Raza' 'Rishi Dhawan' 'Rahul Chahar' 'Harpreet Brar'\n 'Kyle Mayers' 'Jaydev Unadkat' 'Krishnappa Gowtham' 'Avesh Khan'\n 'Mark Wood' 'Ravi Bishnoi' 'Khaleel Ahmed' 'Mukesh Kumar'\n 'Chetan Sakariya' 'Axar Patel' 'Kuldeep Yadav' 'Bhuvneshwar Kumar'\n 'Fazalhaq Farooqi' 'Washington Sundar' 'T Natarajan' 'Adil Rashid'\n 'Umran Malik' 'Trent Boult' 'KM Asif' 'Jason Holder'\n 'Ravichandran Ashwin' 'Yuzvendra Chahal' 'Navdeep Saini' 'Mohammed Siraj'\n 'Reece Topley' 'Akash Deep' 'Harshal Patel' 'Karn Sharma'\n 'Michael Bracewell' 'Glenn Maxwell' 'Jason Behrendorff' 'Arshad Khan'\n 'Jofra Archer' 'Piyush Chawla' 'Cameron Green' 'Hrithik Shokeen'\n 'Krunal Pandya' 'Yash Thakur' 'Ben Stokes' 'Moeen Ali' 'RS Hangargekar'\n 'Anrich Nortje' 'Mitchell Marsh' 'David Willey' 'Shahbaz Ahmed'\n 'Suyash Sharma' 'Deepak Hooda' 'Amit Mishra' 'Aiden Markram'\n 'Rovman Powell' 'Sandeep Sharma' 'Murugan Ashwin' 'Sisanda Magala'\n 'Dwaine Pretorius' 'Kumar Kartikeya' 'Lockie Ferguson' 'Marco Jansen'\n 'Mayank Markande' 'Mohit Rathee' 'Wayne Parnell' 'Riley Meredith'\n 'Tilak Varma' 'Mustafizur Rahman' 'Lalit Yadav' 'Akash Singh'\n 'Maheesh Theekshana' 'Kuldeep Sen' 'Adam Zampa' 'Mohit Sharma'\n 'Kagiso Rabada' 'Matthew Short' 'Andre Russell' 'Vijaykumar Vyshak'\n 'Wanindu Hasaranga' 'Yudhvir Singh Charak' 'Arjun Tendulkar'\n 'Duan Jansen' 'Noor Ahmad' 'Matheesha Pathirana' 'Naveen-ul-Haq'\n 'Liam Livingstone' 'Ishant Sharma' 'Kulwant Khejroliya' 'Anukul Roy'\n 'Nitish Rana' 'Mayank Dagar' 'Marcus Stoinis' 'Jayant Yadav'\n 'Rahul Tewatia' 'David Wiese' 'Vaibhav Arora' 'Kuldip Yadav'\n 'Gurnoor Brar' 'Ayush Badoni' 'Harshit Rana' 'Akeal Hosein'\n 'Abhishek Sharma' 'Josh Hazlewood' 'Mahipal Lomror' 'Akash Madhwal'\n 'Kartik Tyagi' 'Raghav Goyal' 'Tristan Stubbs' 'Mohsin Khan'\n 'Swapnil Singh' 'Vivrant Sharma' 'Obed McCoy' 'Chris Jordan' 'Joe Root'\n 'Yudhvir Singh' 'Glenn Phillips' 'Praveen Dubey' 'Nitish Reddy'\n 'Himanshu Sharma']\n"
"Teams  ['Gujarat Titans' 'Chennai Super Kings' 'Kolkata Knight Riders'\n 'Punjab Kings' 'Lucknow Super Giants' 'Delhi Capitals'\n 'Sunrisers Hyderabad' 'Rajasthan Royals' 'Royal Challengers Bangalore'\n 'Mumbai Indians' 'sunrisers Hyderabad']\nCity  ['Ahmedabad' 'Mohali' 'Lucknow' 'Hyderabad' 'Bengaluru' 'Chennai' 'Delhi'\n 'Guwahati' 'Kolkata' 'Lucknow ' 'Mumbai' 'Jaipur' 'Dharamsala']\n\n"}]

Step 2 — Define OpenAI Functions: In the response, we asked the OpenAI functions for the textual question, and the query to extract answers from the database. Additionally, since we needed to print the wrong options too, had created a field to ask for the answer category.

functions = [
    {
        "name": "generate_question",
        "description": "Generates the answer by running the query in dataset",
        "parameters": {
            "type": "object",
            "properties": {
                "question_prompt": {
                    "type": "string",
                    "description": "Enter a question where the answer is either a batsman, bowler, team or a integer"
                },
                "pandas_query": {
                    "type": "string",
                    "description": "Enter a valid pandas query that is compatible with mentioned schema and gets the answers to the above question",
                },
                "answer_type": {
                    "type": "string",
                    "description": "This should be one of [batsman, bowler, team, out_by]",
                }
            },
            "required": ["question_prompt", "pandas_query", "answer_type"],
        },
    }
]

Step 3 — Process response and post the quiz: Basis the response, we created wrong answers and shuffled them with the right ones, for the user to see it.

Tools that we tried but didn’t end up using in Production:

1. Vercel: 1-click deployment for front-end applications

   • It was easy to do the frontend hosting on Vercel but it was non-trivial to add any logic/backend — tried spending a few hours on “python functions” that they enable, but not have much luck in getting anything live there.

2. LangChain: Wrapper on top of LLMs, designed to build complex products

   • We explored it to try the “memory” feature -- if OpenAI could statefully remember our data schema, it could cut our per-call cost significantly. But we couldn’t get our heads around it completely (turns out it’s so complex that a lot of folks aren’t able to use it).

3. Guardrails: Guardrails help get structured responses

   • We were planning to use LangChain+Guardrails (Couldn’t see Guardrails support for LangChain memory) but LangChain never worked so we didn't use Guardrails either.