Fit a time line to a sound (karaoke)?

[Steve Denney]

Hi, I'd like to create a website where words on a page are under-lined (or highlighted in some way) as they're spoken. It's for ESL students (specifically in remote communities in Australia)--helps improve reading skills.

The sound files will be the entire tract (not single words).

I’ve previously used flash to do this. Flash, of course, displays the sound as volume changes along a time-line (much like Audacity) and although it’s great for animation it proved an extremely tedious job to manually fit the word-on-page animation to the spoken words. Furthermore the students then go on to various tests/games, all nightmarish to program in Flash (but a delight with livecode).

I could, perhaps, use a combination of both: i.e. animation (the karaoke) with flash and exercises in livecode, but if I can somehow fit sound to a timeline in rev I could speed up the karyoke no end.

I’ve toyed with slowing the sound and trying to note the timing of each spoken word. You have the length of the original sound so it gives you a proportional representation of the duration of the word along the timeline – next was to be: hilite word word_number for so many secs.

Failed miserably, in fact, my efforts were laughable.

Anyone have any ideas?
Steve

[bn]

Hi Steve,

you could have a look at callbacks. Callbacks is a list of a) a time, b) a command

When the track plays the player will trigger the command at the given time. You could use this to hilite the words. Of course you would have a list with the times. You could do this in a little script that you record the currentTime when the word is spoken.
Should work.

Kind regards

Bernd

[SparkOut]

Callbacks indeed - and take a look at these very articles:

http://www.runrev.com/developers/lesson ... callbacks/

http://lessons.runrev.com/spaces/lesson ... -Callbacks

[Steve Denney]

Wow, exactly what i'm after by the look of it. I'll report back when i get time to actually do some work on this.
Thanks very much
Steve

[Steve Denney]

Hi, this is the promised report-back (since I may forget if I leave it too long). I've only just begun to test the callbacks concept and may not continue with it.
I found the best way (only way, really) to fit the callback times to the sound file was to use a program like Adobe soundbooth (Audacity would do at a pinch) which provides the elapsed playing time when a word's sound begins.
The relationship between actual elapsed playing time and the callback 'times' is:
callback = (the timeScale of the player) * elapsed time (then adjust number format to match)
It's unfortunately a little more complex when applied, there's a lag, alwaysBuffer helps a little, but i still ended up fudging the values.
Still, if the lag is consistent for long tracts--here's hoping, yet to test this, and how it works in a standalone on the web--I may use callbacks instead of flash.
Thanks again for the help
Steve

[Steve Denney]

Alrighty, since I appear to be sticking around (sadly, I only tend to visit this forum when I have to) here's a further update.
I've made a small demo stack to test callbacks (for myself) and to show the people up north (which is, in fact, far south for many of you). No long tracts (that's yet to be determined), but otherwise it pretty much works as advertised. It's for schoolchildren on a remote community. Any comments, criticisms most welcome,
Steve
http://archers.net.au/Downloads/Bens_Run.html

[Klaus]

Hi Steve,

very nice revlet!

But why didn't someone take a look at the stack for the childrens sake before uploading?
"Cloze Game" on the first card, come on!


Best

Klaus

[Steve Denney]

Heh, Klaus, didn't think of that. Cloze would confuse them no end. Poor little kids have a hard enough time with English forced upon them as it is. Actually, it used to say 'play game'. I changed it to make it sound more formal--like, I will update the story book apps that went to that community (making them karaoke as well) but I plan to pitch the concept to a tertiary institution that currently wants to use flash for adult ESL.
Steve

[Klaus]

Hi Steve,

I'm not sure you understood me, I was complaining about the wrong spelling of "cloZe"


Bet

Klaus

[Steve Denney]

Oh
Well, actually it's spelt right. It's a form of grammar exercise where a word is removed from the text & the students have to guess what the word should be. But it does sound like 'Close' and the kids wouldn't know what a 'Cloze' exercise was... (probably the guy who invented it)--so it'll just confuse them.
BTW you also drew my attention to the fact that the quit button (at least as presently scripted) doesn't do anything in a revlet.
Steve

[Klaus]

Hi Steve,

AHA!
But "quit" does indeed something in a Revlet, it will make the Revlet unresponsive!


Best and happy easter

Klaus

[Steve Denney]

Apparently so. Not what I want, of course, so it'll be going, along with the confusing btn label.
I can't seem to get the sound on the first story page to play on openCard on the first visit. You can see the the 'Listen' btn's clicked but it just refuses to play first time around. It plays if you return to the page though.
The title page is an embedded sound and works just fine, but the story text is done with players (for the callbacks). Is this something you (or anyone else) have seen before?
It might be my fault, there's lots of legacy code behind the scenes. This story-book cloze app idea has been around since hypercard (with many different stories and versions). I've just kept updating it over the years. So it's possible there's some compile error I'm not spotting. If it's not something anyone else has noticed I'll have a closer look at all the scripts.
And thanks very much for all your comments, Klaus (I've just realised I haven't thanked you yet).
Steve

[Steve Denney]

Further to last comment...
I still have no idea why the sound refused to play 1st time through on the 1st text page but have come up with a workaround. I put a player on the previous page (the credits) and now it works. Twilight zone.