Keywords​

Section 001, coding exercises, speech-to-text/dictation software, handwriting detection, AI

Project Abstract​

This document proposes a tool to help professionals and students entering the development workforce practice whiteboard-style interview questions. This product will generate interview-style coding questions and prompt the user to reason out their solution through text and voice dictation. Then, it will generate feedback from the user's answer and thought process to provide them with potential areas of improvement and additional problem-solving techniques.

High Level Requirement​

This app will:

• Provide practice whiteboard interview questions.
• Interpret the user's written input and spoken input.
• Give feedback to the user after attempting a problem.
• Interpret the user’s solution to the problem and analyze it for correctness and optimization.
• Detail specifically where the user can improve.

Conceptual Design​

For ease of access and simplicity of code, I believe this would best be done as a web app with optimization for tablets where users can actually write out their code with a stylus or their finger. This would ideally make the product available across both mobile and desktop platforms and help with accessibility if the user is unable to use any specific type of device. I would propose JS/React for the frontend with a Python backend for processing.

Background​

This project will ideally make use of various open source software in order to meet project requirements. For speech to text dictation, Open AI could be leveraged, as well as for the language processing. This product hopes to improve on existing technologies used to test coding aptitude for interviews such as LeetCode or Codewars which simply use unit testing to determine if a solution is correct or incorrect, rather than testing the thought process of the user which is a major part of interview whiteboard questions.

Required Resources​

This project would require background knowledge of large language models and research into the most efficient uses for this specific problem. Additional research into how best to implement handwriting detection and pseudocode detection would additionally be needed. Additionally, members of the team would need access to a tablet with a stylus for testing purposes.

Web App​

To start using Whiteboard Assistant, visit the web app here: Whiteboard Assistant

Notes:

• Currently, the app is not supported on Firefox. Please use a supported browser such as Chrome, Edge, or Safari.
• For the best experience, use a laptop/tablet device with a stylus pen.

How to Build​

Requirements​

1. Have npm or yarn.
2. Have API keys from OpenAI, Firebase, and iink-ts.
3. Have a browser that supports speech recognition from mdn web docs, to see if your browser supports speech recognition, you can check here.
4. Clone this repository:

git clone https://github.com/capstone-projects-2024-fall/project-whiteboardflow

API Keys​

Create a .env file in the whiteboardflow-frontend directory and add your API keys here. The .env file should look like this:

REACT_APP_FIREBASE_API_KEY=your-firebase-api-key
REACT_APP_FIREBASE_AUTH_DOMAIN=your-firebase-auth-domain
REACT_APP_FIREBASE_PROJECT_ID=your-firebase-project-id
REACT_APP_FIREBASE_STORAGE_BUCKET=your-firebase-storage-bucket
REACT_APP_FIREBASE_MESSAGING_SENDER_ID=your-firebase-messaging-sender-id
REACT_APP_FIREBASE_APP_ID=your-firebase-app-id

Create another .env file in the whiteboardflow-backend directory. Then, add your OpenAI API key in the file:

OPENAI_API_KEY=your-openai-api-key

Create a file named server-configuration.json in the whiteboardflow-frontend\public directory. Then, add your MyScript API key in the file:

{
    "applicationKey": "your-myscript-application-key",
    "hmacKey": "your-myscript-hmac-key"
}

Run Backend​

Navigate to the backend directory:

cd whiteboardflow-backend

Optional: Create and start a virtual environment:

python3 -m venv [your-virtual-environment-name]
source venv/bin/activate

Install the required packages:

pip install -r requirements.txt

Run the server:

python server.py

Run Frontend​

Navigate to the frontend directory:

cd whiteboardflow-frontend

Install the required packages:

npm install

Run the server:

npm start

Usage​

To use the application, navigate to http://localhost:3000 in your browser. You will be prompted to sign in with Google. Once you sign in, you will be able to practice interview questions and attempt to solve them through handwritten input. The application will then analyze your response and provide feedback on your solution.

Features​

Handwriting Analysis​

• Ability to analyze handwritten input to understand and interpret problem-solving strategies and logic.

Microphone Integration for Speech Detection​

• Incorporates speech recognition to allow verbal commands and explanations, enhancing user interaction and input versatility.

AI-Powered Feedback and Performance Tracking​

• Provides real-time, intelligent feedback on the user's problem-solving techniques and solutions.
• Features a comprehensive analytics page that tracks and displays progress in problem-solving efficiency and accuracy.
• Delivers personalized insights into strengths and developmental areas, aiding in targeted skill improvement and ensuring a tailored learning experience.

Expanded Problem Library​

• Extensive collection of coding and estimation problems ranging across various difficulty levels and programming languages.

Customizable Interface​

• Includes a Dark Mode setting to reduce eye strain and improve visibility during extended use.

Security Enhancements​

• Integration with Google account for secure and streamlined user authentication.
• Uses Firebase for secure backend services, including data storage and handling, ensuring that all user information is protected with the latest security standards.

Known Bugs​

Apple Pencil Compatibility​

• Users may experience intermittent loss of pen strokes when using the Apple Pencil.

Session Timeouts​

• On some devices, users might encounter "session timed out" errors occasionally.

Whiteboard Interface Adjustment​

• Occasionally, resizing the problem area on the whiteboard interface by dragging (not using buttons) may fail to adjust properly.

Tool Button Hover State​

• After using certain tools on the whiteboard (e.g., eraser, undo), the hover state on buttons may not disappear as expected.

Initial Problem List Loading​

• Some users may experience longer than usual loading times when accessing the problem list for the first time.

Collaborators​

Dean Roos

Ben Stephenson

Vince Lukban

Renxuan Yao

Dhruvil Patel