Luke Janitch’s Work Log

Week 1 Log:

The augmented reality project has a simple goal in mind, with potential modularity and modifiable aspects. This goal being a simple speech to text device that can calculate who, and where human speech is originating from. Using this information (collected via a multitude of sensors) the device will be able to output the spoken speech into a readable transcript and display it on, or nearby, the source of the identifiable noises.

This past week has consisted of a singular AR team meeting in which we decided on the device’s hardware specifics. Through research and comparisons, as a team, we made final choices on the specific microphones, development board, camera, and temperature sensors needed for the project. We lightly touched on the final idea of the project and mainly emphasized on the goal for the semester. Which is to correctly interconnect our hardware along with our software in order to create a crude prototype. This prototype will consist of a temporary outside display in order to validate that all systems onboard the device are correctly operating, as well as for debugging purposes. It was touched on that the goal for next semester is to clean up the prototype and find a way to make the device comfortably wearable as well as powered in a convenient way. 

Attached below is a crude rendering of the end of semester goal we have set forward to achieve as a team:

As for the plans for next week I plan on starting, or completing, the following:

• Research and identify what software will be used.
• Identify possible compatibility issues with all prospective software.
• Research TDOA techniques.
• Search for information regarding all the interfacing hardware and possible connectivity issues.
• Reconnect with the team and figure out what their next steps will be, as well as inform them of mine.

Week 2 Log:

Week 2 progressed a bit slower in terms of progress on the project due to shipping times for the hardware. However as a team we did meet and discuss some possible issues and concerns with the software side of the project. We brainstormed possible issues and voiced alternate options and routes to accomplish the same end goal. As for next week it is appearing that we will:

• Get hands on with the ordered hardware and brainstorm connectivity between each piece.
• Continue to identify possible compatibility issues with all prospective software.
• Continue research TDOA techniques.
• Work together with the team in order to figure out what the next steps will be after we discuss more on hardware in person.

Week 3 Log:

For week 3, the majority of our hardware required for the project arrived, specifically the Pi 4 board. We collaborated and discussed about different possible ways to run our software on the Pi 4. Additionally the team struck conversation about of the operating speed of OpenCV (an open source AI/machine learning software) and the possible delays it could cause while operating in tandem with the other software’s on the Pi 4. The search of another open source machine learning software was suggested. As a team we also set the weighting of our project goals as well. Basing the weights off of how crucial each specific goal was to the overall project scope. Required for this weeks submission is a design flowchart that can accurately represent the operation of our product:

As for next week we plan to:

• Continue with hands on the hardware and brainstorm connectivity between each piece.
• Continue to identify possible compatibility issues with all prospective software.
• Continue research TDOA techniques.
• Continue working together with the team in order to figure out what the next steps will be after we discuss more on hardware in person.

Week 4 Log:

Week 4 consisted of the group furthering our progress on the project tampering around with Rasbian, our OS of choice originally for the project. We as a team quickly back tracked to an install of Ubuntu instead due to compatibility issues regarding speech to text functionality. We eventually got speech to text to work on the PI 4 but as of right now it works very slowly. We will continue to improve our method throughout this next week.

• Further develop our speech to text process to reduce time and increase efficiency.
• Determine how to implement the actual project microphones so that the PI can use the intended input
• Perhaps begin work on interfacing the PI camera as well.
• Work together with the team in order to figure out what the next steps will be.

Week 5 Log:

During week 5 our group prepared for critical design review, and constructed our presentation for our project so that those attending would be able to criticize and encourage critical thinking pertaining to the specifics of the augmented reality glasses. We as a group have been seemingly behind compared to other projects that have started this semester. I believe this to be as a result of the hardware we ordered is now on back order. Due to this we have been delayed quite a bit in terms of what progress we should be achieving. We as a group have sorted out the back order problem and now await the shipping of our hardware. Next week we plan to:

• Receive the ordered hardware
• Begin physical implementation of the project
• Begin to move what progress we have finished thus far onto a more powerful machine to achieve better results and to aid in debugging
• Work together with the team in order to figure out what the next steps will be.

Week 6 Log:

This week our hardware finally arrived. We sorted through said hardware, and discovered that our 4 channel audio adc was much different in terms of dimensions from what we were expecting. Due to its much smaller size we have begun to brainstorm a new way of implementing the adc. Additionally, we have decided to begin designing our own PCB’s for the project in order to implement all of our collective hardware. Luc has decided to lead the development on said PCB’s with hopes of designing a small, long, but thin board in order for the board to be easily implemented into the temples of our AR glasses. We talked a little about power for the boards and how might we power the device in general, but it seems that this topic will be touched on much more next meeting as very little time was spent on this topic. Next week we plan to:

• Begin PCB Design, and development
• Brainstorm and collectively agree on a method of power delivery for the project
• Begin to move what progress we have finished thus far onto a more powerful machine to achieve better results and to aid in debugging
• Work together with the team in order to figure out what the next steps will be.

Week 7 Log:

The end of semester crunch has begun to set in this week. As such, progress has been continuing along at a much faster rate pertaining to the project. Luc has been furiously at work developing and designing our custom PCB for our MEM’s microphones as well as our 4 audio input ADC (shown below). We are currently running into minor errors with the design through KiCad, but overall it is beginning to appear that we will have our custom board in the next upcoming two weeks. Additionally Stephen has been hard at work designing the rough draft of the glasses that the hardware will eventually end up connected to. It appears that the integration of software will be coming up this next week and that some work will have to be started on the camera, and microphones in order to get them working together in tandem.

For next week we plan to:

• Continue PCB Design, and start PCB debugging
• Brainstorm and collectively agree on a method of power delivery for the project
• Finish up the rough draft of the glasses design
• Begin work on software integration with the PI camera, and MEM’s microphones
• Work together with the team in order to figure out what the next steps will be.

Week 8 Log:

This Week our custom PBS’s were completed and delivered, additionally our capacitors and miscellaneous hardware was also delivered. We have started the process of soldering the electrical components of the the glasses onto their corresponding PCB’s. Luc has decided in the meantime to design an amplifier circuit for the MEM’s microphones to make debugging and testing the fully soldered boards an easier process. I have begun tinkering with the PI in order to get the camera working, and to brainstorm ways to implement image recognition software so that the camera and the software can interface together to determine when a human is speaking.

• Begin amplifier Design, and development
• Brainstorm and collectively agree on a method of power delivery for the project
• Begin interfacing the PI camera and image recognition software
• Work together with the team in order to figure out what the next steps will be.

Week 9 Log:

Throughout the rest if this week the team has been busy making pivotal decisions, we have decided to forgo the hand soldering of the PCB components, and to go with a solder mask instead for ease of construction of the boards. Additionally we have decided to also switch to different MEM’s mics which allow for input on the front of the mic rather than the backside of the mic, this will allow for better audio quality as well. Additionally, the amplifier circuit for the MEM’s PCB has been progressing and we will soon be able to test out the functionality of the current microphone setup. We have decided to continue with the external PC so we can retain processing speeds that we require for the project. Currently the main PCB, separate from the MEM’s PCB’s, is also currently under development as well.

For next week we plan to:

• Begin interfacing the PI camera and image recognition software
• Brainstorm and collectively agree on a method of power delivery for the project
• Finish up the rough draft of the glasses design
• Begin work on software integration with the PI camera, and MEM’s microphones
• Work together with the team in order to figure out what the next steps will be.

Week 10 Log:

This week consisted of the development of the main PCB that is to interface directly with the MEMS PCB via a four channel audio ADC in tandem with an audio amplifier circuit. This board, since development, has been printed and milled for our team in order to begin testing the operational status of all the hardware that has been selected for the rough draft of the final product. Once we have validated the operational status of our hardware we can then begin to focus on the software aspect side of the project more heavily. With hopes of making decent progress on this end before the end of the semester.

For next week we plan to:

• Begin interfacing the PI camera and image recognition software
• Brainstorm and collectively agree on a method of power delivery for the project
• Finish up physical implementation of the hardware and test operational status
• Begin work on software integration with the PI camera, and MEM’s microphones
• Work together with the team in order to figure out what the next steps will be.

Week 11 Log:

This week consisted testing the two main boards and their soldered components. We spent about two hours in the plexus lab with multimeters determining the validity of each resistor, capacitor, and inductors connections as well as using magnifying tools to determine if any of the solder jobs on each component had bridged a connection causing undefined behavior. It appeared that both of our main boards that were tested appeared to have some amount of small amount of issues related to soldered components and solder bridges.

For next week we plan to:

• Begin interfacing the PI camera and image recognition software
• Brainstorm and collectively agree on a method of power delivery for the project
• Begin work on software integration with the PI camera, and MEM’s microphones
• Work together with the team in order to figure out what the next steps will be.

Week 12 Log:

Week 12 consisted mainly of the development of the facial recognition software, which allows for a camera to determine if an individual is speaking or not based off of slight movements in their mouth region. This is achieved by utilizing open CV to identify frontal face regions and from there search the lower half of a face region to identify a persons mouth. Additionally, the hardware underwent more troubleshooting and error correction this week as well.

For next Semester: