At Drexel University’s Digital Media program, our students are pushing
the boundaries of technology and creativity. In a recent DIGM Master’s
course assignment Dheeraj Mantha explored how artificial intelligence (AI) can
transform responsive web development. His project, integrating AI-driven sound
classification with HTML, CSS, and JavaScript…
At Drexel University’s Digital Media program, our students are pushing
the boundaries of technology and creativity. In a recent DIGM Master’s
course assignment Dheeraj Mantha explored how artificial intelligence (AI) can
transform responsive web development. His project, integrating AI-driven sound
classification with HTML, CSS, and JavaScript, showcases the exciting
potential of combining no-code AI tools with web technologies.
The Project: A Sound-Responsive Web Experience
Dheeraj set out to answer a compelling question:
How effectively can a
sound-based AI model, trained with Google’s Teachable Machine,
classify audio inputs like button clicks and keyboard typing, and reflect
those predictions in real-time on a responsive website?
With no prior coding
experience in web development, he aimed to create a website that listens to
real-world sounds—background noise, keyboard typing, or PS5 controller
button presses—and responds by changing its background color and displaying
the detected sound.
Using a browser-based workflow, Dheeraj trained an AI model to recognize
three distinct audio classes, exported it to TensorFlow.js, and integrated
it into a webpage built with HTML, CSS, and JavaScript. The result? A
dynamic interface that shifts colors in real-time: a neutral theme for
background noise, a blue pulsing theme for keyboard typing, and a
yellow-orange shaking theme for controller presses.
Tools and Workflow
Dheeraj leveraged three key tools to bring his vision to life:
-
Google Teachable Machine: This no-code platform allowed Dheeraj to
train a sound classification model by recording audio samples directly in
the browser. Its simplicity and TensorFlow.js export feature made it ideal
for beginners. -
TensorFlow.js: Running entirely in the browser, this library
enabled real-time audio processing without complex backend setup, ensuring
fast and responsive predictions. -
Replit: As an online IDE, Replit provided a seamless environment
for coding and testing the webpage, with live previews to streamline
development.
Despite his lack of coding experience, Dheeraj used ChatGPT to generate the
necessary HTML, CSS, and JavaScript code, adapting it to integrate the AI
model. He faced challenges, including bugs in Teachable Machine’s
recording process and Replit’s preview functionality, but resolved
them through persistence, browser restarts, and refined AI prompts.
Challenges and Triumphs
The journey wasn’t without hurdles. Teachable Machine occasionally
froze during audio recording, requiring multiple retries. Replit’s
preview button failed at times, forcing Dheeraj to test the website locally.
Additionally, Chrome’s persistent microphone permission requests posed
a usability issue. However, by refreshing tools, refining code with ChatGPT,
and tweaking browser settings, Dheeraj overcame these obstacles to deliver a
functional prototype.
The website successfully differentiated between sounds, though the
animations (pulsing and shaking effects) were less dynamic than hoped.
Still, the color changes and real-time text updates provided clear visual
feedback, making the interface engaging and interactive.
Why It Matters
Dheeraj’s project highlights the power of accessible AI tools in web
development. For a beginner with no coding background, the ability to train
an AI model and integrate it into a responsive webpage is a testament to the
democratization of technology. His work also opens doors to real-world
applications, such as:
-
Accessibility: Sound-based controls for websites, enabling interaction
without touch or mouse input. -
Interactive Installations: Dynamic interfaces for gaming or portfolio
websites that respond to environmental sounds. -
Education: Tools for classrooms to trigger UI actions based on ambient
noise detection.
Looking Ahead
Dheeraj plans to enhance the project by adding more sound classes (like
claps or voice commands), improving animations, and exploring Replit’s
AI code generator. He also envisions logging user interactions to analyze
sound patterns over time. These ambitions reflect the iterative spirit of
Drexel’s Digital Media program, where students are encouraged to
experiment and innovate.
A Bright Future for AI in Web Development
This project underscores how AI tools like Teachable Machine and
TensorFlow.js empower students to create sophisticated, responsive web
experiences without deep technical expertise. For Dheeraj, the process was a
crash course in problem-solving, UI design, and the potential of AI-driven
interactivity. His success inspires us all to explore the intersection of AI
and web development.
Stay tuned for more innovative projects from our Digital Media students as
they redefine what’s possible!