Synced actions with audio?

Monkey Forums/Monkey Programming/Synced actions with audio?

Soap(Posted 2013) [#1]
Does anyone know for sure if Monkey has enough by default to reliably sync actions with how long audio has been playing on a timeline? We have done our own tests and they are inconclusive.

Like with any rhythm based game the beats you hit line up with a song.


muddy_shoes(Posted 2013) [#2]
Monkey can't offer exact equivalence across targets for audio latency (or other low-level functions). Some platforms might be fine, others not.


skid(Posted 2013) [#3]

Does anyone know for sure if Monkey has enough by default to reliably sync actions with how long audio has been playing on a timeline?



I doubt it will matter how long the music has been playing. The question is on which targets does MusicState return an honest value so you can timestamp when exactly the music began playing out the speakers (after network, decompression lead times and system latency are taken into account).


Midimaster(Posted 2013) [#4]
in my DUETT project ( http://www.monkeycoder.co.nz/Apps/app.php?id=182 ) I recognized (and you can hear it) that the HTML5 target has big problems in starting sounds at a certain point. Sometimes it sounds exact sometimes not! We recognized differences of 100msec. I seems not to be a fixed latency, but a hickup, that sometimes happens.

The Android target is much better.

For long audios I can conclude, that a already running sound never broke on any target. So you only have the problem of max 100msec offset at the start moment.


Samah(Posted 2013) [#5]
@Midimaster: The Android target is much better.

Huh? Android is terrible. The getCurrentPosition() method on MediaPlayer can't be trusted at all (I've seen differences up to 600ms), and the latency with playing sound samples is atrocious.
http://code.google.com/p/android/issues/detail?id=3434

On the other hand, I've found the HTML5 audio to be quite good. Check out the Diddy storyboard example. I can't vouch for its latency since I haven't tested that, but it has to be more reliable than Android.


Soap(Posted 2013) [#6]
Seems to be the death of a rhythm based game which relies on correct timing in relation to the music then. :(


Samah(Posted 2013) [#7]
@Soap: Seems to be the death of a rhythm based game which relies on correct timing in relation to the music then. :(

Which is why there aren't many of them, and partly why I gave up on my osu! port.


Midimaster(Posted 2013) [#8]
thats exactly what we trying to do: rhythm based games. But I still believe in browser developers. They decided to improve the audio drivers.

On my Android target (a small, slow Samsung with 2.3) I can say, that it is good enough.

And if CPU speed rises, those problems getting smaller over device generations.


Soap(Posted 2013) [#9]
Yes, I think maybe later this year even it may be more viable. Things change fast.


Gerry Quinn(Posted 2013) [#10]
Maybe you could make a rhythm-based game using sounds instead of music. What I mean is that the music track would be broken down into short sounds which are assumed to fire at the time they are played.


Soap(Posted 2013) [#11]
That would work for tracker based music, but there would still be problems if you wanted to do anything complex.

The plan b we are taking is to divorce the patterns from the music completely. And if that doesn't work out to being fun we have a plan c too.

Maybe later this year if things are different we will risk having them in sync - or at least closer to being more viable. Chrome auto updates and that's the only HTML5 browser we care about, so when it's fixed it will be fixed for everyone, and Android devices break over time and need to be replaced so eventually everyone will be on newer Android versions which also have the problem fixed.


Midimaster(Posted 2013) [#12]
@gerry
I'm only talking about music we diveded into sounds and try to fire them in the right moment. Exactly here we have the problems of different latency from sound to sound. So a beat of four sounds fired at...

000msec-250msec-500msec-750msec-1000msec
...sometimes reaches the ear at...
010msec-333msec-510msec-888msec-1010msec

do you see the always changing latency?
+10msec +83msec +10msec +133msec +10msec

What we can recognize is, if you fire 5 sounds at the same moment, they all have the same latency and reach the ear together.

What about chrome browsers. Does somebody have experience with sound latency on it? I always test on Firefox.


Soap(Posted 2013) [#13]
We were testing on Chrome and there was up to 100ms latency with playing audio on localhost, and Chrome lied about the audio actually being played - or at least the Monkey language doesn't account for possible latency and says the audio is playing instantly when it is not. I should look into that. It may be that Monkey assumes it is playing instantly but not that the browser actually is playing instantly, and modifications could be made to make tracking more accurate.