Difference between revisions of "Solipsist Development"

From Robert-Depot
Jump to: navigation, search
(→‎Timeline)
(→‎Open Questions)
Line 24: Line 24:
 
**see [[#Code | Code]] section below.
 
**see [[#Code | Code]] section below.
 
'''<nowiki>*</nowiki>I do not have these items yet.'''
 
'''<nowiki>*</nowiki>I do not have these items yet.'''
−
==Open Questions==
 
−
*Where else do we do this sort of projection and anthropomorphism? (projecting psychology or attributing intention to non-intelligent systems)
 
−
 
 
==Timeline==
 
==Timeline==
 
'''Week 1 - 2'''
 
'''Week 1 - 2'''

Revision as of 17:21, 8 June 2011

Proposal - Solipsist

I have been working with voice recognition technologies and Mel Bochner's text 'Serial Art, Systems, Solipsism', developing a device for performance and exchange between human and computer. The device consists of a microphone, speech recognition system, software, and a receipt printer. The format of the conversation is an dialog between human voice and printed receipts in text--the system transcribes (and validates) what it hears in terms of the words it knows. As is characteristic of voice recognition, face recognition, and other kinds of machine perception that operate within explicitly defined or statistically trained spaces of "perception", this is a solipsistic system, "denying the existence of anything outside the confines of its own mind" (Bochner, 1967). This character of the solipsist is one Mel Bochner evoked to describe the autonomy and denial of external reference in minimalist sculpture of the 1960s, but which I find particularly appropriate to describe current "smart" technologies--ultimately the agency of the system still comes down to whatever agency the programmers have embedded in it, a sort of ventriloquism. This idea of a closed, narrowly parameterized space of perception in the machine is an interesting model for (and contrast to) issues of language, vocabulary, and free expression in humans--an exploration I intend to pursue through this project.

There are multiple challenges in developing this piece as a performance/installation. The first and most concrete is to get a baseline speech recognition system working. I have implemented this in the fall using the Sphinx-4 speech recognition library in Processing and Java. I have also acquired a receipt printer, ribbon, and paper, and can control its printing behavior through a serial interface. Details are in the Code section. This is the "proof of concept".

The development of roles for two characters in this piece--the system and the participant--is necessary to create the kind of encounter I have in mind. On the one hand, I would like this project to investigate the strengths and limitations of the speech recognition technology through viewer interaction, and on the other hand I would like to create a psychological investigation which highlights our human propensity to project psychology onto inanimate things and to attribute intention to them. Explicit attention in constructing the roles of both performers (human and machine) and in framing the situation will tease out some of the interesting ideas in both of these domains.

Finally, I need to address sonic properties of the piece and its time course as a composition. The voice of the viewer as they speak into the microphone is one sound source in the system, and the receipt printer has a very assertive (and retro) dot-matrix-ey sound as it prints out text and cuts rolls of paper. I need to make some decisions about how to use the sounds of the speaker and the printer over the course of the piece. Also, will I add in additional sound sources such as more printers, pre-recorded voices, voices of past participants, or processed sounds? There are additional possibilities here for rhythmic exchanges between the percussive sound of the printer and the speaker, for long pauses, silences, and repetitions. Additionally, I need to establish some overall arc for the piece--does an encounter with the system travel through to one pre-ordained conclusion? Are there multiple branching possibilities that change depending on what the viewer says and how they respond to the printouts? Finally there is a relationship to be explored between speech as text and speech as sound--a parallel to the roles of printing as text and printing as sound. The fundamental distinctions between text, sound, and speech as kinds of communication and expression can be ripe territory for exploration. I suspect that these conceptual and compositional questions will occupy most of my time this quarter and comprise the bulk of the work that I need to do.

The most obvious technical challenges I foresee at this point are the implementation of sound input and pre-processing with supercollider, and interfacing from supercollider to the speech recognition library and receipt printer. As part of this project is a critical investigation of the strengths and limitations of automatic speech recognition (ASR) technology, I intend to get more involved with the internal mechanisms of speech recognition as implemented in the Sphinx-4 library. A more comprehensive understanding of that technology is necessary to figure out how to tweak it and expose its internal character and assumptions.

I will update the weekly Progress section as the quarter continues.

Equipment

  • Hardware:
    • receipt printer
    • microphone
    • motu box.
    • mac mini.
    • desk*
    • desk lamp*
    • sound-proof commercial glazing (windows)*
  • Software:

*I do not have these items yet.

Timeline

Week 1 - 2

Introduction to course and project development.


Week 3 - Proposal - 4/12

  1. Write this.
  2. Meet with Juan.
  3. Find desk and desk lamp for "me vs. the computer" staging of microphone/speech recognition system.

Week 4

Work time.

Week 5 - MILESTONE 1

Working model of each of two one tracks:

  1. Participant speaking to computer.
  2. Computer/printer speaking to itself (feedback loop). Interpreting printer sounds as speech. Or transforming them into speech.

Week 6

Realize that the best approach will combine elements of each of the two tracks above.

Week 7 - MILESTONE 2

  • Experiments with the characterization of the system:
    • Software agent? agency. towards what goals?
    • Interruptions.
    • Unexpected responses.
    • Basic state modeling (emotional states, psychological states).
    • Basic drive modeling (for novelty, entertainment, activity, rest, conversation on certain topics).
  • Attention to possible choices of text.

I imagine these two go hand in hand--that the choice of particular texts will lend much of the character to the piece.

Week 8

Have others experience the system, try it out.

Week 9 - MILESTONE 3

Near-final form, near-final realization.

Viewer interaction tests.

Week 10

Final changes, improvements, last minute blitz.

Presentation

Progress

Week 2

get system running again

Week 3

  • Got the system running again. The hard to find OS X driver for my Airlink101 AC-USBS (Serial Adapter) was actually available from Prolific: http://www.prolific.com.tw/eng/downloads.asp?id=31. I think the airlink device must have the Prolific PL-2303 USB to I/O Port Controller inside. Finding a driver probably wouldn't be difficult if I bought some more recently manufactured usb-to-serial adapter, if people still manufacture those sorts of things. I need a serial adapter because the epson receipt printer has some old school serial connectivity on the back.
  • Met with Juan.

Week 4

Week 5 - Milestone 1

Done:

  • working model
  • bidirectional OSC communication between processing/supercollider
  • some planning

Pictures (on my computer)

To Do - Conceptual:

  • end state? (criteria for cut)
  • changing identities?
  • learn from participants? (if so, what model/vocabulary am I matching against, e.g. WSJ4K, TI_DIGITS, custom vocab?)
  • bored behavior? (when no-one is in the room)

To Do - Technical:

  • folding camp/travel table.
  • speech classifier in supercollider
  • save out audio files of speech sound
  • responder. state modeling? what language?
  • sound isolation possibilities. multi-pane commercial glazing would be my preference, though it is expensive I'm sure.

Week 6

Pocketsphinx:

Language Models:

Sphinx Website Stuff:

Articles:

Week 7 - Milestone 2

In Class Discussion: 

put a coin to purchase the receipt
occupying roles
master script: stelios
learning from its own output: nico
that's what I said: juan
more than just the filter: juan
some analysis of the text
reconfigure the text
what's the thing about the receipt printer?: nico
coin-op justifies the receipt.  commercial transaction
record of a transaction
oracle, coin-op oracle: nico
trimpin, coin-op sculpture: juan
automatic poetry generator
ask people to engage poetically with it, record them
the idea of the residual, the parts/things that people say that are outside of the system (it's closed language-field)
site-specificity: administrative  scout out sites with speech rec system.
not be so one-dimensional, analyze tonality, intonation, rate.  profiling affect.

Week 8

Josh P. suggests white rabbit code (libsamplerate) to do my supercollider -> 16KHz (sphinx) downsampling. The following three libraries can be built and installed:

  1. pkg-config 0.23 builds on OS X: http://mac.softpedia.com/progDownload/pkg-config-Download-46896.html
  2. libsndfile: http://www.mega-nerd.com/libsndfile/#Download
  3. libsamplerate: http://www.mega-nerd.com/SRC/download.html

Convert audio file to 16-bit little-endian audio raw:

  • ffmpeg -i solipsism1.wav -acodec pcm_s16le -ac 1 -ar 16000 solipsism1_pcm_s16le.wav
  • cmu07a.dic

Later in the week:

Week 9 - Milestone 3

To Do:

  • finish connecting up components (e.g. SC to pocketsphinx and back)
  • enable self-modifying grammar/Language Model
  • Updated pocketsphinx-osx code to accept command line arguments:
    • run recognition with a grammar: ./pocketsphinx-osx-continuous -jsgf solipsist.gram -infile solipsism1_pcm_s16le.wav
    • run recognition with a custom language model: ./pocketsphinx-osx-continuous -dict 9316.dic -lm 9316.lm

Week 10

Allright! I made much progress on my speechtool supercollider code. (http://svn.roberttwomey.com/supercollider/supercollider/speechtool.scd)

New features:

  • It is relatively sensitive to sound levels and does a decent job of detecting possible speech above background noise.
  • It correctly saves out raw time-stamped audio files to ~/Sounds/SpeechFiles/ of all sounds detected and sent to pocket-sphinx
  • It calls sndfile-resample to resample 44.1KHz audio files to 16KHz for pocket-sphinx.
  • It runs recognition with pocketsphinx and prints results to the post window in the format Recognized: ....
  • It saves a text logfile in ~/Sounds/SpeechFiles/logs with the DateStamped .wav file name and the recognized string.
  • It now works with a JSGF grammar rather than a statistical language model. Example command line options:
  • With an SLM from online tool:
    • ./pocketsphinx-osx-continuous -dict /Users/rtwomey/Documents/dxarts463_sp11/mmpi-2/9316.dic -lm /Users/rtwomey/Documents/dxarts463_sp11/mmpi-2/9316.lm
  • With an SLM from local tools:
    • ./pocketsphinx-osx-continuous -lm /Users/rtwomey/Documents/dxarts463_sp11/mmpi2_slm/mmpi2.arpa

Info on default models:

  • HMM (acoustic?) hub4wsj_sc_8k (hmm/en_US/hub4wsj_sc_8k)
  • Dictionary: cmu07a.dic, 133436 words (lm/en_US/cmu07a.dic)
  • Filler dictionary: noisedict (hmm/en_US/hub4wsj_sc_8k/noisedict)
  • LM... either hub4.5000.DMP or wsj0vp.5000.DMP, I am guessing. (?)

To Do:

  • Field recordings with baseline vocabulary and language models. (I think this is the WSJ5K)
  • Test with different LMs or Grammars
  • Make more decisions.

Finals Week

Rhyming + Homophonic sounds:

Building a Grammar:

Building a Language Model:

Generating a Dictionary:

  • http://cmusphinx.sourceforge.net/wiki/tutorialdict
  • from command line: perl make_pronunciation.pl -tools /Users/rtwomey/code/cmusphinx/trunk/logios/Tools/ -dictdir /Users/rtwomey/code/cmusphinx/trunk/logios/Tools/MakeDict/lib/dict -words /Users/rtwomey/Documents/dxarts463_sp11/mmpi2_slm/mmpi2.tmp.vocab -handdict NONE -dict results.dic

Language World:

Language Models:

Grammars:

Final

Tar File

portable_solipsist.tar

contains supercollider code, os x binaries, data files, etc. see README inside for more info.

Report

coming wed. am.

Code

Processing

Supercollider

Pocketsphinx command-line Recognizer in OS X

Build a Language Model Online

Build a Language Model Locally

  • see week 10 and final week above... not yet done

How To Compile on OS X

Basic CMU Sphinx-4 Automatic Speech Recognition (ASR) library info:

Compiling pocketsphinx as a universal static lib on OS X

  • make x86_64 version of libsphinxbase:

cd sphinxbase-0.7
./configure
make

  • copy resulting libsphinxbase.a file from /sphinxbase-0.7/src/libsphinxbase/.libs/ to libsphinxbase.x86_84.a in temp directory
  • make x86_64 version of libspocketsphinx:

cd pocketsphinx-0.7
./configure
make

  • copy resulting libpocketsphinx.a file from /pocketsphinx-0.7/src/libpocketsphinx/.libs to libpocketsphinx.x86_64.a file in temp directory
  • make i386 versions of libsphinxbase:

export CFLAGS="-arch i386" 
export LDFLAGS="-arch i386"
cd sphinxbase-0.7
make clean
./configure
make

  • copy resulting libsphinxbase.a file from /sphinxbase-0.7/src/libsphinxbase/.libs/ to libsphinxbase.i386.a in temp directory
  • make i386 versions of libpocketsphinx:

cd pocketsphinx-0.7
make clean
./configure
make

  • copy resulting libpocketsphinx.a file from /pocketsphinx-0.7/src/libpocketsphinx/.libs to libpocketsphinx.i386.a file in temp directory
  • combine files with lipo

lipo -create -output libsphinxbase.a libsphinxbase.x86_64.a libsphinxbase.i386.a
lipo -create -output libpocketsphinx.a libpocketsphinx.x86_64.a libpocketsphinx.i386.a

Building logios LM tools

 export CFLAGS="-I/usr/include/malloc"
./configure
make
sudo make install

References

Retrieved from "http://wiki.roberttwomey.com/index.php?title=Solipsist_Development&oldid=4593"