Question generation using state-of-the-art Natural Language Processing algorithms
Questgen AI
Try advanced question generation models for free: https://questgen.ai/
Questgen AI is an opensource NLP library focused on developing easy to use Question generation algorithms.
It is on a quest build the world’s most advanced question generation AI leveraging on state-of-the-art transformer models like T5, BERT and OpenAI GPT-2 etc.
Online course and blog
🚀 Our online course that teaches how to build these models from scratch and deploy them
Currently Supported Question Generation Capabilities :
1. Multiple Choice Questions (MCQs) 2. Boolean Questions (Yes/No) 3. General FAQs 4. Paraphrasing any Question 5. Question Answering.
Simple and Complete Google Colab Demo
1. Installation
1.1 Libraries
pip install git+https://github.com/ramsrigouthamg/Questgen.ai
pip install git+https://github.com/boudinfl/pke.git
python -m nltk.downloader universal_tagset
python -m spacy download en
1.2 Download and extract zip of Sense2vec wordvectors that are used for generation of multiple choices.
wget https://github.com/explosion/sense2vec/releases/download/v1.0.0/s2v_reddit_2015_md.tar.gz
tar -xvf s2v_reddit_2015_md.tar.gz
2. Running the code
2.1 Generate boolean (Yes/No) Questions
from pprint import pprint
import nltk
nltk.download('stopwords')
from Questgen import main
qe= main.BoolQGen()
payload = {
"input_text": "Sachin Ramesh Tendulkar is a former international cricketer from India and a former captain of the Indian national team. He is widely regarded as one of the greatest batsmen in the history of cricket. He is the highest run scorer of all time in International cricket."
}
output = qe.predict_boolq(payload)
pprint (output)
Show Output
'Boolean Questions': ['Is sachin ramesh tendulkar the highest run scorer in '
'cricket?',
'Is sachin ramesh tendulkar the highest run scorer in '
'cricket?',
'Is sachin tendulkar the highest run scorer in '
'cricket?']
2.2 Generate MCQ Questions
qg = main.QGen()
output = qg.predict_mcq(payload)
pprint (output)
Show Output
{'questions': [{'answer': 'cricketer',
'context': 'Sachin Ramesh Tendulkar is a former international '
'cricketer from India and a former captain of the '
'Indian national team.',
'extra_options': ['Mark Waugh',
'Sharma',
'Ricky Ponting',
'Afridi',
'Kohli',
'Dhoni'],
'id': 1,
'options': ['Brett Lee', 'Footballer', 'International Cricket'],
'options_algorithm': 'sense2vec',
'question_statement': "What is Sachin Ramesh Tendulkar's "
'career?',
'question_type': 'MCQ'},
{'answer': 'india',
'context': 'Sachin Ramesh Tendulkar is a former international '
'cricketer from India and a former captain of the '
'Indian national team.',
'extra_options': ['Pakistan',
'South Korea',
'Nepal',
'Philippines',
'Zimbabwe'],
'id': 2,
'options': ['Bangladesh', 'Indonesia', 'China'],
'options_algorithm': 'sense2vec',
'question_statement': 'Where is Sachin Ramesh Tendulkar from?',
'question_type': 'MCQ'},
{'answer': 'batsmen',
'context': 'He is widely regarded as one of the greatest '
'batsmen in the history of cricket.',
'extra_options': ['Ashwin', 'Dhoni', 'Afridi', 'Death Overs'],
'id': 3,
'options': ['Bowlers', 'Wickets', 'Mccullum'],
'options_algorithm': 'sense2vec',
'question_statement': 'What is the best cricketer?',
'question_type': 'MCQ'}]}
2.3 Generate FAQ Questions
output = qg.predict_shortq(payload)
pprint (output)
Show Output
{'questions': [{'Answer': 'cricketer',
'Question': "What is Sachin Ramesh Tendulkar's career?",
'context': 'Sachin Ramesh Tendulkar is a former international '
'cricketer from India and a former captain of the '
'Indian national team.',
'id': 1},
{'Answer': 'india',
'Question': 'Where is Sachin Ramesh Tendulkar from?',
'context': 'Sachin Ramesh Tendulkar is a former international '
'cricketer from India and a former captain of the '
'Indian national team.',
'id': 2},
{'Answer': 'batsmen',
'Question': 'What is the best cricketer?',
'context': 'He is widely regarded as one of the greatest '
'batsmen in the history of cricket.',
'id': 3}]
}
2.4 Paraphrasing Questions
payload2 = {
"input_text" : "What is Sachin Tendulkar profession?",
"max_questions": 5
}
output = qg.paraphrase(payload2)
pprint (output)
Show Output
{'Paraphrased Questions': ["ParaphrasedTarget: What is Sachin Tendulkar's "
'profession?',
"ParaphrasedTarget: What is Sachin Tendulkar's "
'career?',
"ParaphrasedTarget: What is Sachin Tendulkar's job?",
'ParaphrasedTarget: What is Sachin Tendulkar?',
"ParaphrasedTarget: What is Sachin Tendulkar's "
'occupation?'],
'Question': 'What is Sachin Tendulkar profession?'}
2.5 Question Answering (Simple)
answer = main.AnswerPredictor()
payload3 = {
"input_text" : '''Sachin Ramesh Tendulkar is a former international cricketer from
India and a former captain of the Indian national team. He is widely regarded
as one of the greatest batsmen in the history of cricket. He is the highest
run scorer of all time in International cricket.''',
"input_question" : "Who is Sachin tendulkar ? "
}
output = answer.predict_answer(payload3)
Show Output
Sachin ramesh tendulkar is a former international cricketer from india and a former captain of the indian national team.
2.6 Question Answering (Boolean)
payload4 = {
"input_text" : '''Sachin Ramesh Tendulkar is a former international cricketer from
India and a former captain of the Indian national team. He is widely regarded
as one of the greatest batsmen in the history of cricket. He is the highest
run scorer of all time in International cricket.''',
"input_question" : "Is Sachin tendulkar a former cricketer? "
}
output = answer.predict_answer(payload4)
print (output)
Show Output
Yes, sachin tendulkar is a former cricketer.
NLP models used
For maintaining meaningfulness in Questions, Questgen uses Three T5 models. One for Boolean Question generation, one for MCQs, FAQs, Paraphrasing and one for answer generation.
Online Demo website.
