Interactive Telephony for New Media Arts

Phone for voice and touchtone? Who woulda thunk..?

Although it seems everyone is using SMS, playing video games, listening to music and streaming video on their mobile devices (phone or portable computer?) it turns out that mobile phones are fantastic at transmitting voice over a distance..! Not only that, voice and touchtone are ubiquitous, 100% of people who have a phone have used it to make a phone call. SMS is the next closest use of phones worldwide with around 50% of people having used it.

Although voice and touchtone are heavily used in everyday life, people don't often think of them as an artistic medium or a way to enable interaction with other forms of new media. Because mobile phones are so ubiquitous, always on, always with us and calls that utilize voice and touchtone are easy and familiar, they make an ideal place to experiment and use as controllers.

Examples

Here are some projects that I find illuminating:

Megaphone: Big screen games controlled via touchtone
Bontanicalls: Plants that call you when they need water
Speed Dial: Voice controlled Slot Cars
Dialtones a Telesymphony: Our very own Golan Levin
Blinkenlights: Chaos Computer Club in Berlin, large scale telephone controlled displays
Surprise Dialer: Group initiated voicemail
Payphone Murder Mystery: Interactive narrative using payphones in subway
Payphone Warriors: urban game using payphones
Improv Everywhere Cellphone Symphony: just fun
Community Voicemail: community based voicemail system
Jott: Voice to text notes
GrandCentral: Google Voice
Humor Hotlines: Rejection hotline
Payphone Project: Database of payphones
They Might Be Giants Dial-A-Song

PSTN: Public Switched Telephone Network

The worlds "circuit" switched phone network. Originally dedicated circuits were created between the caller and the receiver and managed by an operator. The audio was transmitted as analog.

Now the only part of the PSTN system (generally) that is still analog is the connection to the home or business from the telephone network. The bulk of the audio transport is digital but still circuit switched (as opposed to using IP routing on the internet).

Telephone System Test Board
Wikipedia: PSTN
Telephone Tribute

VoIP: Voice over IP

VoIP can stand for anything that carries voice over an IP network (such as the internet). Recently bandwidth and latency have gotten to the point where the internet can reliably be used to transmit voice.

There are many types of VoIP but probably the most widely used standard is SIP (Session Initiation Protocol). SIP is similar to SMTP (Mail Transport Protocol) in that a message is sent from one party to another. In VoIP applications, the SIP message will normally define what ports, codecs and transport mechanisms are to be used to send the audio.

Asterisk: Bridging PSTN with VoIP

Asterisk is a PBX (a Private Branch Exchange). Commonly used in businesses, a PBX allows a company to have multiple extensions, offer voicemail, handle queues of incoming calls and so on. Generally a PBX will have a standard PSTN line to the telephony company and may have IP connections to handsets.

Asterisk is an open source fully functional PBX with both PSTN and VoIP capabilities. It is also highly configurableable and can be scripted with a variety of languages.

Because Asterisk is VoIP capable, it doesn't nessecarily need a PSTN connection to a regular phone company. It can simply receive VoIP calls from the internet and utilize a third party to offer PSTN connectivity.

There are a slew of companies who offer this service, typically referred to as PSTN gateways. These companies will register a number with the large carriers and when calls come in or are sent out will do the translation from PSTN to VoIP and VoIP to PSTN.

The asterisk install that we will be working with is setup in this manner and is using a company called DID World Wide to provide incoming call service.

Asterisk Console

I find that the Asterisk console, although command line based is the best way to get to know what is going on with Asterisk.

Unfortunately, explaining the unix command line is a bit more than the scope of this workshp but a good place to learn about it is Webmonkey's Enough Unix for your Resume

To enter the CLI (Command Line Interface) for Asterisk (the console), ssh to the server and issue the following command:

/usr/sbin/asterisk -r

You should see something like this:

Asterisk 1.4.10.1, Copyright (C) 1999 - 2007 Digium, Inc. and others.
Created by Mark Spencer 
Asterisk comes with ABSOLUTELY NO WARRANTY; type 'core show warranty' for details.
This is free software, with components licensed under the GNU General Public
License version 2 and other licenses; you are welcome to redistribute it under
certain conditions. Type 'core show license' for details.
=========================================================================
Connected to Asterisk 1.4.10.1 currently running on asterisk (pid = 15927)
Verbosity is at least 3
asterisk*CLI>

Probably the most useful part of having access to the console is being able to debug your application.

When a call comes in you start to see output such as this:

-- Executing Playback("SIP/vanevery-7716", "beep") in new stack
-- Playing 'beep' (language 'en')
-- Executing VoiceMail("SIP/vanevery-7716", "u10@testingtesting") in new stack
-- Playing 'vm-theperson' (language 'en')
-- Playing 'digits/1' (language 'en')
-- Playing 'digits/0' (language 'en')
-- Playing 'vm-isunavail' (language 'en')
-- Playing 'vm-intro' (language 'en')
-- Playing 'beep' (language 'en')
-- Recording the message
-- x=0, open writing:  /var/spool/asterisk/voicemail/testingtesting/10/INBOX/msg0002 format: wav49, 0x8869280
-- x=1, open writing:  /var/spool/asterisk/voicemail/testingtesting/10/INBOX/msg0002 format: gsm, 0x886d080
-- x=2, open writing:  /var/spool/asterisk/voicemail/testingtesting/10/INBOX/msg0002 format: wav, 0x88780c0

You will also see error messages which helps immensly when trying to debug a problematic application.

Dialplan

The asterisk dialplan is what you will be using to construct the logic that occurs when a call comes into Asterisk.

Generally the dialplan is defined in a configuration file called: /etc/asterisk/extensions.conf. By default, a call is routed to the default context which is the section defined by [default].

To make things simpiler for us, especially since many of us will be using the system at the same time, I added the following lines into the extensions.conf file on the server we are using:

[default]
;
; By default we include the demo.  In a production system, you
; probably don't want to have the demo there.
;
;include => demo

#include "cmu_user0_extension.conf"
#include "cmu_user1_extension.conf"
#include "cmu_user2_extension.conf"
#include "cmu_user3_extension.conf"
#include "cmu_user4_extension.conf"
#include "cmu_user5_extension.conf"
#include "cmu_user6_extension.conf"
#include "cmu_user7_extension.conf"
#include "cmu_user8_extension.conf"
#include "cmu_user9_extension.conf"
#include "cmu_user10_extension.conf"
#include "cmu_user11_extension.conf"
#include "cmu_user12_extension.conf"
#include "cmu_user13_extension.conf"
#include "cmu_user14_extension.conf"

exten => _X,1,Goto(s,1);
exten => _X.,1,Goto(s,1);
exten => s,1,NoOp(${CALLERID(NAME)})
exten => s,n,Set(TIMEOUT(digit)=5)
exten => s,n,Set(TIMEOUT(response)=10)
exten => s,n,Set(TIMEOUT(absolute)=600)
exten => s,n,Set(numtimeouts=0);
exten => s,n,Answer();
exten => s,n,Wait(1);
exten => s,n(PlayExtension),Background(vm-extension);
exten => s,n,WaitExten(10);
exten => t,1,Set(numtimeouts=$[${numtimeouts} + 1]);
exten => t,n,GotoIf($[${numtimeouts} >= 4]?999,1);
exten => t,n,Goto(s,PlayExtension);
exten => t,1,Goto(s,PlayExtension);
exten => i,1,Playback(invalid);
exten => i,n,Goto(s,PlayExtension);

exten => 0,1,Goto(cmu_user0,s,1);
exten => 1,1,Goto(cmu_user1,s,1);
exten => 2,1,Goto(cmu_user2,s,1);
exten => 3,1,Goto(cmu_user3,s,1);
exten => 4,1,Goto(cmu_user4,s,1);
exten => 5,1,Goto(cmu_user5,s,1);
exten => 6,1,Goto(cmu_user6,s,1);
exten => 7,1,Goto(cmu_user7,s,1);
exten => 8,1,Goto(cmu_user8,s,1);
exten => 9,1,Goto(cmu_user9,s,1);
exten => 10,1,Goto(cmu_user10,s,1);
exten => 11,1,Goto(cmu_user11,s,1);
exten => 12,1,Goto(cmu_user12,s,1);
exten => 13,1,Goto(cmu_user13,s,1);
exten => 14,1,Goto(cmu_user14,s,1);

This section of code isn't terribly important for us to get started but it is worthwhile to explain.

The first set of lines #include "cmu_user0_extension.conf" specifies that we should include a set of external files. These external files are the ones that you will be editing (as I have specified in the handout you were given).

The second section of code defines what should happen when a call comes into the server. It uses several commands available in the Asterisk dialplan to ask the caller to punch in the extension that they want to go to.

The third section of the code exten => 0,1,Goto(cmu_user0,s,1); defines where the call should go depending on what extension the user enters.

So as explained, we are dynamically including each of your dialplan files into this file when we "reload" asterisk through the web page that was setup. For all practical purposes, you are directly editing that file when you write your own dialplan files.

Let's tackle the syntax of the dialplan language:

Context

A context is a central idea to asterisk dialplans. It defines a section of the dialplan that any incoming or outgoing call is executing. There can be hundreds of contexts defined and logic within those contexts can move calls from one to another.

For instance, when a call comes in via our number, it is put into the context that our SIP service is registered to:

[default]

This context does a couple of things but most important is that it asks the caller for an extension and then uses the "Goto()" command based on the extension that was entered to send the call to another context (the ones we have defined).

exten => 2,1,Goto(cmu_user2,s,1);

If they enter extension 2, it goes into the context that I have defined for that which is "cmu_user2" with the "s" extension for start and to the command with priority 1.

Extensions

Extensions are the means that asterisk uses to match a command within a context. There are a couple of special extensions as follows:

s - The "start" or "special" extension. Where things generally start within a context. More
t - The "timeout" extension. If a Digit or Response timeout has occurred. More
i - The "invalid" extension. If the user enters an extension that isn't matched. More
h - The "hangup" extension. Executes after a call has hungup. More

Normally an extension though is just a number, so if the user enters 2, they will go to that command.

Priority

Within a context and an extension Asterisk uses priorities to determine which command to run.

Generally these go in order from 1 to 2 and so on. The "n" priority is essentially the previous priority + 1 which makes it easier to work with dialplans than specifying each number.

Here is a full dialplan for the "cmu_user0" context:

[cmu_user0]
exten => s,1,Answer();
exten => s,n,Wait(1);
exten => s,n,Background(vm-extension);
exten => s,n,WaitExten(10);

exten => t,1,Goto(s,3); 

exten => i,1,Playback(invalid);
exten => i,n,Goto(s,3);

exten => 1,1,Playback(tt-weasels)
exten => 1,n,Goto(s,3);

exten => 2,1,SayDigit(2);
exten => 2,n,Goto(s,3);

Going through this line by line should help illuminate what is going on here:

The first line, [cmu_user0], specifies that the following lines are are part of the "cmu_user0" context.
The next line, exten => s,1,Answer(); specifies that the first thing, indicated by "1" that should be done in the "s" extension (start), is to "answer" the call.
The next series of lines, all with the format of exten => s,n,Command() indicate what should happen next "n".
The last line in the series, exten => s,n,WaitExten(10);, indicates that the "WaitExten" command should execute. It means that it will wait 10 seconds for the caller to enter an extension. What happens next depends on what extension is entered.

If the user enters "1", this series of commands executes:

exten => 1,1,Playback(tt-weasels)
exten => 1,n,Goto(s,3);

If the user enters "2", this series is executed:

exten => 2,1,SayDigit(2);
exten => 2,n,Goto(s,3);

If the user enters something that isn't expected, like "3", it goes to the "i" (invalid) extension:

exten => i,1,Playback(invalid);
exten => i,n,Goto(s,3);

If the user doesn't enter something in the 10 seconds given, it goes to the "t" (timeout) extension:

exten => t,1,Goto(s,3);

Asterisk Config Extensions - voip-info.org

Applications and Commands

Asterisk has a LOT of commands/applications that you can use. We have already looked at: Answer, WaitExten, Background, Playback, Goto Here are a few more:

Recording Audio:
Records what a user says for a specified period of time.

exten => s,1,Record(/path/to/file.gsm,5,30);

The first argument is the name of the file with the format, after the comma it is the amount of silence detection (in seconds I beleive) and the next argument is the amount of time to record.

Asterisk cmd Record

Playback with Controls:
Plays an audio file with controls.

exten => s,n,ControlPlayback(/path/to/file/toplay,1000,#,*,0,1,2);

The first argument is the name of the file to play, the second the number of milliseconds to move per button push. The last arguments are the buttons for rewind, fast forward, stop, play and pause.

Asterisk cmd ControlPlayback

Monitoring/Recording a Call:
Asterisk can record the audio of a call. By default it saves the audio in two separate files, one for each side of the "conversation".

exten => s,1,Monitor(wav,shawn_monitor);

This command will create two files, both in /var/spool/asterisk/monitor. One will be named shawn_monitor-in.wav, the other shawn_monitor-out.wav.

Asterisk cmd Monitor
Also checkout MixMonitor which mixes the two channels at: Asterisk cmd MixMonitory

Here is an example that pulls a couple of the things we have learned together:

[cmu_user0]
; This is a comment
; Exten "s" is the start extension
; Monitor the call (records to the asterisk_spool/monitor directory)
exten => s,1,Monitor(gsm,shawn_monitor)
; Playback helloworld.gsm from the asterisk_sounds directory
exten => s,n,Playback(helloworld)
; Wait for the user to enter an extension
exten => s,n,WaitExten(10)

; If the push 0, record a new version of helloworld.gsm
exten => 0,1,Record(helloworld.gsm,5,30)
; Go back to the playback command
exten => 0,n,Goto(s,2)

; If they push 1, say their caller id
; The variable is CALLERID(NUMBER)
; You pull the value out with the ${xxxx} syntax
exten => 1,1,SayDigits(${CALLERID(NUMBER)})
; There is also Set for variables and so on

; If they push an "invalid" extension, playback the invalid sound file
exten => i,1,Playback(invalid)
exten => i,2,Goto(s,2)

; If they wait too long, "timeout", hangup..
exten => t,1,Hangup()

A ton more applications/commands (check the alphabetical list)

AGI Scripting

AGI stands for Asterisk Gateway Interface. It is very similar to CGI (common gateway interface) which is one of the first forms of web development. AGI Scripting with Asterisk opens up phone calls to a whole new level, anything you can do with any language can now be done through a phone call.

Calling AGI Scripts

To call an AGI script from the Asterisk dialplan you use the AGI command

		exten => s,1,Answer();
		exten => s,n,AGI(shawn_agi_test.php);

By default Asterisk looks for the AGI script in the agi-bin (/var/lib/asterisk/agi-bin). You can place them anywhere and use the full path: /home/cmu_user/shawn_agi_test.php

Debugging

In the Asterisk administration console you can type: agi set debug to enable AGI debugging. Type agi set debug off when you get sick of it.

Basic AGI Scripting in PHP

I find the open source implementation of AGI in PHP to be pretty nice to work with: PHPAGI on SoureForge. Documentation available as well: PHPAGI Docs.

Here is a very basic example using the PHPAGI library:

#!/usr/bin/php -q
<?PHP		
require('/var/lib/asterisk/agi-bin/phpagi.php');

$agi = new AGI();

$agi->stream_file("vm-extension");
$return = $agi->wait_for_digit(10000);
if ($return['result'] > 0)
{
		$ascii = chr($return['result']);
		$agi->say_number($ascii);
}
?>

Upload, Test and Run

Save the above script to a file, let's call it "easy.php" for now. Using Fetch or another SFTP application upload the script to the "asterisk_agi" in your directory. Log into the server using SSH and navigate to the directory containing the PHP (cd asterisk_agi) and execute the command required to make the file executable (chmod 755 easy.php).

To test that the file runs without error, type the following:

./easy.php

This should start the execution of the file, if it doesn't there is an error that you need to fix. The application should wait indefinitely for input (which it is expecting from Asterisk through STDIN). You can simulate this by hitting return a couple of times. You should see some semblance of the commands that are sent to Asterisk via STDOUT.

The above example just scratches the iceberg of what can be done by interfacing Asterisk with PHP.

Here is an example which pulls the weather from NOAA:

#!/usr/bin/php -q
<?PHP		
	require('/var/lib/asterisk/agi-bin/phpagi.php');

	$agi = new AGI();

	// Predefined Variables
	/*
	$agi->request["agi_request"]
	$agi->request["agi_channel"]
	$agi->request["agi_language"]
	$agi->request["agi_uniqueid"]
	$agi->request["agi_allerid"]
	$agi->request["agi_dnid"]
	$agi->request["agi_rdnis"]
	$agi->request["agi_context"]
	$agi->request["agi_extension"]
	$agi->request["agi_priority"]
	$agi->request["agi_enhanced"]
	$agi->request["agi_accountcode"]
	$agi->request["agi_network"]
	$agi->request["agi_network_script"]
	*/

	/*      
	// Exec any dialplan application/function
	$return = $agi->exec("Application","Options");
	if ($return['result'] == 1)
	{
					$agi->say_number(1);
	}
	*/

	# for a complete list of US cities, go to 
	# http://www.nws.noaa.gov/data/current_obs/ 
	$weatherURL = array();
	$weatherURL[0]="http://www.nws.noaa.gov/data/current_obs/KJFK.xml"; // NYC
	$weatherURL[1]="http://www.nws.noaa.gov/data/current_obs/KAGC.xml"; // Pitts
	$weatherURL[2]="http://www.nws.noaa.gov/data/current_obs/KART.xml";
	$weatherURL[3]="http://www.nws.noaa.gov/data/current_obs/KBGM.xml";
	$weatherURL[4]="http://www.nws.noaa.gov/data/current_obs/KBUF.xml";
	$weatherURL[5]="http://www.nws.noaa.gov/data/current_obs/KDKK.xml";
	$weatherURL[6]="http://www.nws.noaa.gov/data/current_obs/KDSV.xml";
	$weatherURL[7]="http://www.nws.noaa.gov/data/current_obs/KELM.xml";
	$weatherURL[8]="http://www.nws.noaa.gov/data/current_obs/KHPN.xml";
	$weatherURL[9]="http://www.nws.noaa.gov/data/current_obs/KFRG.xml";
	
	
	$continue = true;
	while($continue)
	{
			$agi->stream_file("vm-extension");
			$return = $agi->wait_for_digit(10000);
			//$agi->say_number($return['result']);
			
			if ($return['result'] > 0)
			{
				// chr turns numeric ascii value into actual number or letter
				$ascii = chr($return['result']);
			}
			else
			{
					$continue = false;
			}
	
	
			$weatherPage=file_get_contents($weatherURL[$ascii]); 
	
			$currentTemp = -100;
			if (preg_match("/<temp_f>([0-9.]+)<\/temp_f>/i",$weatherPage,$matches)) 
			{ 
					$currentTemp=$matches[1]; 
			} 
	
	
			if ($currentTemp > -100) 
			{ 
					$agi->say_number($currentTemp);
					//$agi->say_digits($ascii);
			}
			else
			{
					$continue = false;
			}
	}
?>

PHPAGI PHP Library Docs
PHPAGI Site

Creating a Web Service with AGI

One direction we can go is to use PHP and a web server to bridge to other applications that could be running anywhere.

Below is an implementation of an asterisk/php webservice which exposes data from phone calls via the internet:

This PHP script is AGI. Asterisk will execute it when a call comes in:

#!/usr/bin/php -q
<?PHP
require('/var/lib/asterisk/agi-bin/phpagi.php');

$agi = new AGI();

$agi->stream_file("vm-extension");
$return = $agi->wait_for_digit(10000);
while ($return['result'] > 0)
{
        $ascii = chr($return['result']);
        $agi->say_number($ascii);
        file_put_contents("/var/www/html/webservice/data.txt",time() . "," . $agi->request["agi_uniqueid"] . "," . $agi->request["agi_callerid"] . "," . $ascii . "\n",FILE_APPEND);
        $return = $agi->wait_for_digit(100000);
}
?>

Give it a name and upload it to the asterisk_agi directory. In this example I am calling it, bridge.php.

Then in your asterisk dialplan you will need to execute it when a call comes in:

exten => s,1,Answer();
exten => s,n,AGI(bridge.php);

That's it for the asterisk side. Now we need another PHP script to serve the data via the web:

<?
	// The client will send a timestamp if it want's new stuff from the timestamp
	$timestamp = 0;
	if (isset($_GET['ts']))
	{
		$timestamp = $_GET['ts'];
	}

	$data = file_get_contents("data.txt");
	$dataarray = explode("\n",$data);
	if ($timestamp > 0)
	{
		// Send everything from the timestamp forward
		for ($i = sizeof($dataarray) - 10; $i < sizeof($dataarray); $i++)
		{
			$currentline = explode(",",$dataarray[$i]);
			if (sizeof($currentline) > 0)
			{
				if ($currentline[0] > $timestamp)
				{
					echo($dataarray[$i]."\n");
				}
			}
		}
	}
	else
	{
		// Just send the last one
		if (sizeof($dataarray) > 1)
		{
			echo($dataarray[sizeof($dataarray)-2]);
		}
	}
?>

It returns the touchtone data that has come in since the last time the request for the data was made.

The data is returned in this format which is the same format it is saved by asterisk: timestamp,unique_id,caller-id,digit-pressed

Now we can make an app in any language that can make an HTTP request and get this data..

Here is a Processing example: Asterisk AGI which is a modified version of Dan Shiffman's Calling Processing with Asterisk tutorial.

Where to go from here?

A fair number of resources exist for using asterisk in creative ways. Here are a set of links that I think would probably be useful to you:

Redial: Interactive Telephony: Syllabus with weekly notes from course I teach at NYU's ITP
Voip-Info.org: A fantastic wiki dedicated to VoIP and Asterisk
Asterisk: The Future of Telephony (second edition) a great book that takes you through most standard uses of Asterisk. Published with a Creative Commons license and available for free download
A Tutorial for Asterisk on Amazon's EC2 Service
Dan Shiffman's Notes on using Processing with Asterisk - His examples us a Java AGI script instead of PHP (which probably has better performance).
DID World Wide: A good source for incoming call phone numbers world wide.
vanevery@walking-productions.com