Experiments in making music by atrodo
Its once again self reflection time, and this year I have more than usual to reflect on. While I did write one blog post along with a flurry of work at the beginning of the year, the unfortunate reality of the second half of the year slowed progress down significantly.
Continue...Since the start of the year, I’ve been working, on and off, on getting the robots into more specialized lessons. I have a much improved process that allows me to listen to a single robot’s progress, figure out which one is causing issues and make adjustments.
Continue...Another year, and another New Years blog. This year, however, is a bit different since there have been no other blog posts this year. It is a bit representative of the past year as there was not much teaching of robots happening for most of the past year.
Continue...It has been a long year, and in this final week of the year, it is time to take stock about what has happened. As is tradition, I’m also taking a moment to apply this process to Teaching Robots how to Drum.
Continue...Although it’s been a few months since the last post and there has not been much progress to report, I feel like the goal of this blog and project is to keep going until I hit a point that it can be called complete. So in light of that, I have been continuing to upload videos for months now and am working towards improving the entire process. Unfortunately, I’ve been trapped in looking at numbers again and making things works numerical, so it is time to redirect my efforts.
Continue...A few months ago, I was explaining that I was wanting to start uploading files again. I was looking to start soon, but that soon became a few weeks and a few weeks became a few months as caution and trepidation meant that I was hesitant to pull the trigger and start uploading.
Continue...At the beginning of the year, my original plan was to start uploading new videos within the first few weeks of January. And indeed I had all of the robots trained and produced a sample mid-January. The sample was even somewhat respectable and pleasing enough to listen to. The problem, however, was that none of the robots were correctly following the melody. That led to 3 more months of fighting both the fitness functions and the neural network inputs. I am hoping that I’m nearing the end, again, of having good enough to start. I finally feel like I’ve tightened the robot’s ability to follow the melody, and I’m hoping that what comes out will be good enough.
Continue...I started messing around with the code that would become Teaching Robots how to Drum back at the end of 2018. At the end of 2019 I recapped what I had done for the first time, setting the precedent of yearly reflection I now find myself doing. I appreciate the time I take to look back on the past year that I probably would not do otherwise. This includes taking a deep look at all the information; the stats, the blogs, and what has happened in a set period of time.
Continue...It has been more than a few months since I finished reflecting on my robot making music experiment. Since then, I’ve been working towards a few goals to get back to daily video uploads. Even though it’s its taken longer than expected for various reasons, several unrelated to the project, I am almost back to that place of generating daily video uploads.
Continue...It has been 30 days since I began releasing music created by robots. It’s be a very interesting 30 days, and I’ve learned quite a lot doing it. I’ve already made and planned a couple improvements for the existing set of uploads and am also looking at future improvements. With all of that that behind me, I wanted to record briefly what I did do and some ideas and plans I have for improvements going forward.
Continue...Over the past few months, I’ve been steadily working towards releasing what I have taught these Robots. I am pleased to announce, that as of today, I’ve successfully done exactly that. I will be releasing a new song, every day, for the next thirty days to showcase what I’ve taught these robots so far. After that, I am going to evaluate what needs to happen, but right now I’m already excited about the future.
Continue...Recently, I started listening to more melodies and soon discovered that some of the melodies were not nearly as consistently transcribed as I thought. Some songs came out very well done, while others had a more soup-like consistency. I had to take some time again to dive in and find what the problem this time was.
Continue...It’s the end of the year, and I thought I’d take a moment once again to reflect on Teaching Robots again. While it has been a bizarre year, I’m taking a moment to look back at what I’ve done in the past year and how I feel about it. I must admit that today, looking back at what I’ve done I feel proud about what I’ve been able to accomplish this year, despite the set backs the year had to offer.
Continue...While I’d like to think that this project is going well, the unfortunate reality is that it could be going much better. We don’t always want to talk about how things didn’t go according to plan, especially when no one knew there was even a plan at all. That said, I feel like I should write down about one of these well laid plans that no one knew about and how it didn’t go as planned.
Continue...A new bee has entered my bonnet. Actually, it’s an old bee: melody extraction.
As I am working towards generating music, I wanted to play with more ideas to
improve melody extraction. Unfortunately none of these ideas panned out, so I
went looking for new ideas and came across the
Essentia library. I am super excited about finding
this library and being able to use it.
Soon after my last post, I had finished the manual fitness tests and got them all working enough to start listening to some output. The problem became, however, that even though my automatic fitness checks were saying that the output was good, the audio told a different story. While the rhythm was being followed to some degree, it was not nearly as good as the fitness functions said they were; in some cases it was playing notes that did not sound well at all together but saying that it was following the notes of the original melody well.
Continue...After my last post, I was finally able to save my samples back to Ogg once
again, with a good amount of speed. This has enabled me to once again begin
streaming these samples in order to evaluate the robots progress. With my new
found extra speed, I set about realizing my goal, only to find out that there
are more obstacles.
After my journey in making a process to create and manually evaluate samples, I
started working on the actual mechanics of how to get a rating back into the
system. After some initial flawed designs, the new, better design sent me down
a rabbit hole debugging ffmpeg, x264 and libtheora until finally coming to
the point where I can generate and evaluate all the samples I want.
While I’ve been refining my process for generating new samples to train my music making robots, I’ve had to look at a lot of the video I’ve been producing. Right now, I’m making a less than abstract interpretation of the audio. Quite the opposite actually, it’s just a reflection of the music being played. I want to replace it eventually, but that’s a slightly larger than I want to tackle now. While it might seem that coloring each square a pre-chosen color would be simplest, that didn’t seem like much fun. Plus, since not everything will always be ordered in the same way, doing it that way means things are not always colored in the same way. So instead, I’ve decided to color each instrument based on their strings.
Continue...A general problem I’ve known I’ve had to deal with soon is handling the fact that the automatic evaluation, or the fitness function, of how well the neural networks are performing can only take me so far. The fitness code I have now does a good job of keeping clearly wrong things from being used in future neural networks, but it does not do a good job of really evaluating the actual musicality of the music the neural networks are creating.
Continue...After my last big project of writing a melody extraction routine, I took a bit of a break for a couple weeks. Understanding the code and how it works, and especially the blog post related to the work I did there, was a bit draining. Now I’m back to making robots making music.
Continue...As 2019 closes, I wanted to take a moment and reflect on everything I’ve done this year with Teaching Robots. The first commit I made was in late 2018, so most of the development has been done this year. Looking back, I can see that I’ve done a lot of coding and discovery this year.
Continue...After writing the previous two posts, I wanted to take a moment to share an experience I had in hopes that if someone else experiences it as well, this can help them. I had replicated Cooley-Tukey in code and was getting what I knew were accurate results because FFTW was producing the save results.
Continue...This is part 2 of a 2 part series about Discrete and Fast Fourier Transformations. The first part is about the theory and how DFTs work while the second part is about the practical application and code execution to get the frequency data from a sound sample.
Continue...This is part 1 of a 2 part series about Discrete and Fast Fourier Transformations. The first part is about the theory and how DFTs work while the second part is about the practical application and code execution to get the frequency data from a sound sample.
Continue...I discovered some bugs with the fitness functions, the automated verification process used to evaluate the neural networks on basic attributes of each band member like being on beat or following the melody. While the symptoms of the bugs were easy to spot, like rapid notes and not filling the entire time, the causes of these bugs were not obvious at first. I had to dig into the data to discover the issues.
Continue...I have been working on adding more infrastructure around the music generation in order to support diagnosing and improving all the moving parts involved in the process. The first things I did was work on visuals. I added code that will visualize the notes that the melody and each band member plays and turns them into a video. There’s a lot of work there to be done before it’s interesting to watch, but there’s enough there to help me examine the output visually.
Continue...The melody of a song is, to me, the core musical part that really identifies something as a specific song. It’s the line that you hum when its stuck in your head, and it’s the sequence that helps you identify it from a million other songs.
Continue...At the end of the last post I was just wrapping up getting the streaming working. I did some cleanup of that then moved on to some house keeping I had been meaning to do to try and improve the general health of the code base. None of the issues were particularly important, but it provided a nice pause before working on the next thing.
Continue...The feature I’ve been tackling is that I want to add the ability to stream the songs that are created so I can reduce the feedback loop between generating a song and testing it. So with one command I can tell the process to generate it and listen to it immediately without switching programs and reloading it, it’ll just start playing. So I’ve been working on getting more async processing working, but there have been some bumps in the road and some complications, which has been more annoying than I think it should have been. I’ve been using AnyEvent and UV, creating an async model that takes commands from stdin as well IRC and is able to stream all at the same time.
Continue...In my last post, I talked about how the fitness function could eventually come up with a good score to being on the up beat (played half way through a beat). So, as a fun mental exercise, and I promise it wasn’t to prove myself right, I decided to explain how it could have.
Continue...While working through adding the “new band” command I noticed that the drummer was struggling to generate a new instrument I had added. After a some investigation, I discovered that it had broken on the “up-beat” fitness I had just created. Originally, for the drummer, I had an “on-beat” fitness function to make sure that the output was a quick, on the down beat activation. I added several new fitness functions, including one for on the downbeat, on each quarter beat, and each triplet beat. At the same time, I also added one for the up-beat. This one happens exactly between each down beat, so when divided into 12ths, this is suppose to activate on the 6th (when starting at 1) part.
Continue...After last episode’s dilemma, I decided to try and do something on the simple
side. So I started writing a super simple IRC bot. It takes some simple
commands, like !bands and responds with information about it. Soon after I
started to work on the commands, I started exploring how to test the commands
easily. After a quick stint with irssi again, it occured to me that I could just
as easily connect the bot to STDIN as well. So that’s what I did.
I am continuing the march on reducing moving parts and migrating code from the monster script to the database schema. I’ve now reduced much of the code required to produce a song down to a few lines of code:
Continue...A few years ago, for a product that is no longer around, I helped a co-worker craft a fun little function I’ve decided to recreate. The purpose was to create more human friendly code that we could throw up into URLs instead of UUIDs. We use a lot of UUIDs, and I’m a big fan of them. But they are ugly when they are put in URLs. For a lot of our products, that wasn’t a big deal. For this particular product, we wanted to be more consumer conscience and friendly. So we devised a simple idea: generate a short code from the UUID.
Continue...I am just wrapping up a database reorganization that introduces the song table. So far, my monster of a main script has been doing that work, but now that code can finally move into a new home. Took a couple iterations, but I think I’ve settled. Originally the structure was going to just point at the band and generate the wav from that. But then it wouldn’t be repeatability as the neural networks would change over time and produce different outputs. So while I maintain a reference to the band that created the song and each track, I went farther and record the data structure used to generate the actual music.
Continue...I began a journey several months ago to create music through programming. The proposed process has morphed a lot over that time as my original ideas became roadblocked. Road blocked either because of computational, technical, or even resource limitations. As I’m working on it, I’ve decided I should chronicle what I’m doing and what I’ve learned. So I’ve decided to start. But, since I’m in the middle of the process, and I’m lazy, I’m not to back fill the information I’ve already learned all at once. However, I will try and revisit those things as time goes on, I think there’s at least a couple things I’ve learned that is worth sharing.
Continue...