HOME | DD

Description
I was stumped on a motion project the other day. As I mentioned to ~GM34 I was staring at the screen for about a half hour without being able to do a darn thing. Simply put, I was stuck and I didn't know why; so I took the dog out for a walk and the reason occurred to me. So when I came back home, I put the other project on hold and started this one. I learnt somethings here and I would like to share that knowledge with you.

I've watched a lot of MMD videos and have played with, used or adapted a lot of MMD motion data. Some of it is excellent work by real masters at this sort of work. But with very few exceptions, they lack the "emotional" impact that a live human performer can convey.

I wanted to know if it was possible to create an animation using an MMD character that can convey the emotions of a song in the manner that real live human divas can do.

So I went and watched a whole bunch of videos of real divas singing - not idols but divas. There is a difference. What is it about the performances that someone like Celine Dion, for example, can convey but which in general seems to be lacking in MMD performances? What is it that they do, that keeps us mesmerised to the screen?

If you look at Celine Dion perform, she doesn't really move around that much yet she doesn't need to. She doesn't in general move around that much because she's human and needs to breathe. Miku Hatsune doesn't need to breathe. She can belt out a song at an amazing speed and dance at the same time. Because she's a Vocaloid, not human. Yet she is a parody of a human. Miku would not be as popular as she is if she didn't remind us of us.

Now, if I did a motion trace of Celine Dion performing, I can tell you it would be a really boring video to watch no matter how good the music maybe, simply because she doesn't move much. So what works for the real live Celine will not in general work for Miku. So what is the solution?

So I went back and plowed through a whole series of singing animations by the masters of animated song and dance. The Disney studios. And I think I worked out what it is that they do to make animated characters convey real performance even though in actual fact all they are doing is lip-sync. It can be summed up in a single word: DETAILS.

Disney character "breathe". You have to watch very carefully as it's very subtle. But there seems to be another secret. Their movements, although very life like, are caricatures. They're actually over the top in a very subtle and precise way. If a Disney character moves, it's for a reason. If Celine Dion moves it's for a reason. This much they have in common. Both only move when they have to.

But there's another common feature. Face and hands. The reason why we can watch Celine Dion for all 5 mins of her performance is what she's doing with her face and to a lesser extent her hands. Disney characters when they move, put a subtle emphasis on their movements. Let me try to explain this in MMD terms. Say for example you want your MMD model to raise her arm. Most people would be inclined to just manipulate the arm bone. But that's not how we raise our arms, the movement actually starts at the shoulder and in some instances, even the spine is affected.

When Celine Dion sings, the sound doesn't just come out of her mouth, it comes out of her lungs. When you watch her, you can see this. And this was something I missed out on in my earlier animations. I can make the model's lip sync almost perfect, but when looking back at it, it's just lip sync. The model isn't really singing. To properly animate a singing MMD model, you have got to consider what's happening to her "virtual" lungs. It's very subtle, but the Disney characters have this, which is why they look like they're really singing.

Every movement that an animated character makes must have a reason and there are a lot of good reasons why they should move. Also these movements should be precise, hierarchical and for emphasis. So for example a blink is just a quick eye movement, but if the character deliberately closes the eyes other things happen such as the eye brows moving down.

The other thing about real divas is that they have "personality". Likewise, our MMD characters must also convey personality when they perform. How is this achieved? Celine Dion singing the same song as, say for example, Whitney Houston won't do it in the same way. Each one will have their personal stamp on their performance (whatever these may be). So how do we do this with MMD models? In a nutshell, it's done through the details that you put into the animation and this goes back to my earlier point about movements being precise and for a reason.

Now at this point I can probably write an entire chapter on this, but I know you won't want to read that so I'll just detail briefly what I did for this video. If you watch it, it'll make more sense.

First off, I made a deliberate choice to choose a song that demands "performance". I've only used a minutes worth as this was really just to test out what I've learnt. The song was sung by 70's diva Donner Sommers and is a classic. Now Donner Sommers has a very deep rich voice and that does not work for the model I chose to use for this video. The model is Rhea by Kanihira and is of a girl in her mid to late teens. She would not have the voice qualities of Donner Sommers for a variety of biological reasons so the first challenge was to rehash the original recording in Audacity. The clip sample I worked from was very poor quality so it did need a lot of work to create, but I won't go into the details. Simply put I changed the octave and then seperated the stereo tracks and reworked them individually and put them back together again.

Then I watched Donner Sommer's performance over and over again to try and understand her interpretation of this material. Right off the bat, a motion trace was out of the question. So the next step was to create an interpretation for Rhea, and this was done in two ways. First I imagined what she is like as a character (much as I do when I create characters for my comedic skits). Then I went through the whole motion sequence myself (thank god the neighbors weren't watching) whilst listening to the song clip over and over again until I understood exactly what Rhea's body would be doing.

I then played the song in Windows Media player with MMD opened and did several dry runs using the Rhea model to understand how to move the model. Then I took my copy of Rhea to the PMD editor and made some functional edits to make it easier to animate, especially, her hands.

The next step was to build the lip sync data and this was done in two passes. First pass was just the lips. When I was happy with that I then did her facials. Specifically the eyes. Rhea has, like most anime girls, very big eyes. This does not work with this song. So very careful attention was paid to what she did with her eyes and in actual fact, she very seldom has her eyes fully opened. Even her blinks were precisely planned and timed. Then the eye movements were detailed with complementing eyebrow facials. This is important to convey the proper emotions. Once this was done I created the VMD data for this portion.

Animating Rhea was done in MMD 7.39. This was done in several passes. The first pass was to block in her torso, left arm (which was holding the mic) and her head. Once this was done, the second pass was to animate what she did with her right arm which is used to emphasize the emotions conveyed by the words of the song. The third pass was to add details such as turns of the head. The fourth pass was to add the actual eyeball movements. This is was done so that she actually only makes eye contact with the viewer once towards the beginning of the sequence. Sort of a "look at me" hook which was emphasised by making her clear the hair away from her face at the beginning. Remember what I wrote earlier about every movement having a reason? After that Rhea doesn't look at the viewer because I want the viewer to look at her. Then I reviewed the entire sequence several times and made tiny corrections or added additional details as needed such as detailed animation of the fingers.

There is no fancy camera work in this sequence which lasts for a little over 1 minute. It doesn't need it. What I did do was to set up the scene using the Rule of Thirds MME tool and the camera's focal length. After that, Rhea's performance was expected to work on it's own.

Lighting used is actually backlighting. This was emphasized by correctly placing the skydome and using the correct MME shaders. Colors are very vibrant and rich and the stage selected is toned down. I wanted viewers to look at Rhea, not the scenery.

How well all of this works, well you decide. Please check out the video at the link given below. If you read this far, I hope you have found some useful ideas within the wall-of-text I wrote here.

Video Link: [link]

CREDITS
Rhea: Kanihira (version used here was edited for improved functionality)
Falcon Crest Stage - [link]
Skydome - ~kaahgome
Microphone - [link]
MME - Over Ray Shader / Diffusion 7 / Rule of Thirds
Lighting and camera set up - trackdancer.deviantart.com
Motion - ~Trackdancer
I'm sorry that I cannot offer this motion for download as it was done on a non-standard bone model which means that the data will not transfer gracefully to other models.
Music - Remixed clip from Donner Summer's 1978 rendition of the Jimmy Webb classic.
The version used here was modified to make it sound like it is being sung by a younger singer and is used herein under Fair Use guidelines.

Software Used: MMD 7.39, MOGG Project Face and Lips, Audacity 2.01, Photoshop 7, SketchUp 8.0

Description
I was stumped on a motion project the other day. As I mentioned to ~GM34 I was staring at the screen for about a half hour without being able to do a darn thing. Simply put, I was stuck and I didn't know why; so I took the dog out for a walk and the reason occurred to me. So when I came back home, I put the other project on hold and started this one. I learnt somethings here and I would like to share that knowledge with you.

I've watched a lot of MMD videos and have played with, used or adapted a lot of MMD motion data. Some of it is excellent work by real masters at this sort of work. But with very few exceptions, they lack the "emotional" impact that a live human performer can convey.

I wanted to know if it was possible to create an animation using an MMD character that can convey the emotions of a song in the manner that real live human divas can do.

So I went and watched a whole bunch of videos of real divas singing - not idols but divas. There is a difference. What is it about the performances that someone like Celine Dion, for example, can convey but which in general seems to be lacking in MMD performances? What is it that they do, that keeps us mesmerised to the screen?

If you look at Celine Dion perform, she doesn't really move around that much yet she doesn't need to. She doesn't in general move around that much because she's human and needs to breathe. Miku Hatsune doesn't need to breathe. She can belt out a song at an amazing speed and dance at the same time. Because she's a Vocaloid, not human. Yet she is a parody of a human. Miku would not be as popular as she is if she didn't remind us of us.

Now, if I did a motion trace of Celine Dion performing, I can tell you it would be a really boring video to watch no matter how good the music maybe, simply because she doesn't move much. So what works for the real live Celine will not in general work for Miku. So what is the solution?

So I went back and plowed through a whole series of singing animations by the masters of animated song and dance. The Disney studios. And I think I worked out what it is that they do to make animated characters convey real performance even though in actual fact all they are doing is lip-sync. It can be summed up in a single word: DETAILS.

Disney character "breathe". You have to watch very carefully as it's very subtle. But there seems to be another secret. Their movements, although very life like, are caricatures. They're actually over the top in a very subtle and precise way. If a Disney character moves, it's for a reason. If Celine Dion moves it's for a reason. This much they have in common. Both only move when they have to.

But there's another common feature. Face and hands. The reason why we can watch Celine Dion for all 5 mins of her performance is what she's doing with her face and to a lesser extent her hands. Disney characters when they move, put a subtle emphasis on their movements. Let me try to explain this in MMD terms. Say for example you want your MMD model to raise her arm. Most people would be inclined to just manipulate the arm bone. But that's not how we raise our arms, the movement actually starts at the shoulder and in some instances, even the spine is affected.

When Celine Dion sings, the sound doesn't just come out of her mouth, it comes out of her lungs. When you watch her, you can see this. And this was something I missed out on in my earlier animations. I can make the model's lip sync almost perfect, but when looking back at it, it's just lip sync. The model isn't really singing. To properly animate a singing MMD model, you have got to consider what's happening to her "virtual" lungs. It's very subtle, but the Disney characters have this, which is why they look like they're really singing.

Every movement that an animated character makes must have a reason and there are a lot of good reasons why they should move. Also these movements should be precise, hierarchical and for emphasis. So for example a blink is just a quick eye movement, but if the character deliberately closes the eyes other things happen such as the eye brows moving down.

The other thing about real divas is that they have "personality". Likewise, our MMD characters must also convey personality when they perform. How is this achieved? Celine Dion singing the same song as, say for example, Whitney Houston won't do it in the same way. Each one will have their personal stamp on their performance (whatever these may be). So how do we do this with MMD models? In a nutshell, it's done through the details that you put into the animation and this goes back to my earlier point about movements being precise and for a reason.

Now at this point I can probably write an entire chapter on this, but I know you won't want to read that so I'll just detail briefly what I did for this video. If you watch it, it'll make more sense.

First off, I made a deliberate choice to choose a song that demands "performance". I've only used a minutes worth as this was really just to test out what I've learnt. The song was sung by 70's diva Donner Sommers and is a classic. Now Donner Sommers has a very deep rich voice and that does not work for the model I chose to use for this video. The model is Rhea by Kanihira and is of a girl in her mid to late teens. She would not have the voice qualities of Donner Sommers for a variety of biological reasons so the first challenge was to rehash the original recording in Audacity. The clip sample I worked from was very poor quality so it did need a lot of work to create, but I won't go into the details. Simply put I changed the octave and then seperated the stereo tracks and reworked them individually and put them back together again.

Then I watched Donner Sommer's performance over and over again to try and understand her interpretation of this material. Right off the bat, a motion trace was out of the question. So the next step was to create an interpretation for Rhea, and this was done in two ways. First I imagined what she is like as a character (much as I do when I create characters for my comedic skits). Then I went through the whole motion sequence myself (thank god the neighbors weren't watching) whilst listening to the song clip over and over again until I understood exactly what Rhea's body would be doing.

I then played the song in Windows Media player with MMD opened and did several dry runs using the Rhea model to understand how to move the model. Then I took my copy of Rhea to the PMD editor and made some functional edits to make it easier to animate, especially, her hands.

The next step was to build the lip sync data and this was done in two passes. First pass was just the lips. When I was happy with that I then did her facials. Specifically the eyes. Rhea has, like most anime girls, very big eyes. This does not work with this song. So very careful attention was paid to what she did with her eyes and in actual fact, she very seldom has her eyes fully opened. Even her blinks were precisely planned and timed. Then the eye movements were detailed with complementing eyebrow facials. This is important to convey the proper emotions. Once this was done I created the VMD data for this portion.

Animating Rhea was done in MMD 7.39. This was done in several passes. The first pass was to block in her torso, left arm (which was holding the mic) and her head. Once this was done, the second pass was to animate what she did with her right arm which is used to emphasize the emotions conveyed by the words of the song. The third pass was to add details such as turns of the head. The fourth pass was to add the actual eyeball movements. This is was done so that she actually only makes eye contact with the viewer once towards the beginning of the sequence. Sort of a "look at me" hook which was emphasised by making her clear the hair away from her face at the beginning. Remember what I wrote earlier about every movement having a reason? After that Rhea doesn't look at the viewer because I want the viewer to look at her. Then I reviewed the entire sequence several times and made tiny corrections or added additional details as needed such as detailed animation of the fingers.

There is no fancy camera work in this sequence which lasts for a little over 1 minute. It doesn't need it. What I did do was to set up the scene using the Rule of Thirds MME tool and the camera's focal length. After that, Rhea's performance was expected to work on it's own.

Lighting used is actually backlighting. This was emphasized by correctly placing the skydome and using the correct MME shaders. Colors are very vibrant and rich and the stage selected is toned down. I wanted viewers to look at Rhea, not the scenery.

How well all of this works, well you decide. Please check out the video at the link given below. If you read this far, I hope you have found some useful ideas within the wall-of-text I wrote here.

Video Link: [link]

CREDITS
Rhea: Kanihira (version used here was edited for improved functionality)
Falcon Crest Stage - [link]
Skydome - ~kaahgome
Microphone - [link]
MME - Over Ray Shader / Diffusion 7 / Rule of Thirds
Lighting and camera set up - trackdancer.deviantart.com
Motion - ~Trackdancer
I'm sorry that I cannot offer this motion for download as it was done on a non-standard bone model which means that the data will not transfer gracefully to other models.
Music - Remixed clip from Donner Summer's 1978 rendition of the Jimmy Webb classic.
The version used here was modified to make it sound like it is being sung by a younger singer and is used herein under Fair Use guidelines.

Software Used: MMD 7.39, MOGG Project Face and Lips, Audacity 2.01, Photoshop 7, SketchUp 8.0