Real-time text analysis to determine new elements What we learned Working on an unsolved machine learning problem (sentence simplification) Making a web application, with a variety of machine learning and non-machine learning techniques. But despite our challenges, we enjoyed the opportunity and are grateful for that. Also, since none of us was a UI developer, we faced particular challenges in styling the app the way we initially intended to. One of the other challenges was to patiently debug the cloud functions which we had no experience with. We couldn't explore the APIs of other services like Scale because of time constraints. We ended up developing a custom pipeline for bullet-point generation based on 'Part-of-speech' and 'Dependency analysis'. Text summarization is very difficult - there may be powerful algorithms to turn articles into paragraph summaries, there is essentially nothing on shortening sentences into bullet points. Once it receives the details back from the model, it broadcasts the new elements that need to be added to the slides and the front-end JavaScript adds the content to the slides. When the server receives a relevant WebSockets message, it creates a connection to our socket server running the machine learning model and sends information about what the user has been saying to the model. Unfortunately, the machine learning prediction could not be run within the web app itself, so we had to put it into its process and thread and send the information over regular sockets so that the website would work. In addition to the WebSockets portion of our project, we had to use internal socket communications to do the actual text analysis. This summarization is performed using custom Part-of-speech analysis, which starts at verbs with many dependencies and works its way outward in the dependency tree, pruning branches of the sentence that are superfluous. When the user is talking “normally,” we attempt to summarize their speech into bullet points. We do this by searching through sentence dependency trees to find short sub-phrases (1-5 words optimally) which contain important words and verbs. Using Part-of-speech analysis, we determine which parts of a sentence (or sentences) would best serve as a keyword for the message. We then used Facebook’s Deep Learning text classification library, FastText, to train a custom neural network that could perform text classification. So we gathered a repository of sample sentences from news articles for “no” examples, and manually curated a list of “yes” examples. TOPIC ANALYSISįundamentally we needed a way to determine whether a given sentence included any relevant information that could be conveyed through text. The Text To Speech is captured whenever a user records their audio, and when they let go the aggregated text is sent to the server over WebSockets to be processed. We use Microsoft Azure's speech-to-text API to process audio on the microphone of the device. Our various NLP tasks are facilitated through a custom back-end API in Python and Microsoft Azure's speech-to-text functionality. To build Textify, we used React-native as a front-end framework, coupled with Firebase for our database functioning. By tapping at any of the keywords one can navigate quickly to the timestamp of the message. It also utilizes topic detection, to give summarized information to the message's recipient. Using sentiment analysis, Textify gives visual feedback of the other person's emotions in an audio message. Textify is an audio messaging app that automatically shortens long voice messages to a more manageable length, and provides a transcription of the message for those of us who are too busy to listen to the entire thing! □ We thus wanted to explore and enable new ways of communication, only available now because of recent developments in computation and machine learning. I'm sure we all regularly face the problem of receiving uncomfortably long and unclear voice messages from our friends because they're too lazy to type it out! Even with today's robust technologies, communication still happens to be one of the bottlenecks in human interaction.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |