This discussion is archived
5 Replies Latest reply: Aug 2, 2010 8:51 AM by captfoss RSS

Multi-TargetDataLine PCM Downmixing to Single Stereo WAV File

843802 Newbie
Currently Being Moderated
Hello Java Sound Gurus:

I am capturing PCM-44100Hz-16bit-2Channel audio simultaneously from 2 microphones (TargetDataLines).

This yields 4 channels that have be mixed down to 2 channels and written-out to a WAV file (no compression, no change to the AudioFormat.)

I do capture data alright and it does get written to the WAV file, HOWEVER, there are two problems with resultant WAV file:

1. Even though I capture 20 seconds of audio, the WAV file header always shows only two seconds of audio, even though the "data" chunk conrtains all 20 seconds of audio! (File is the right size, too.)

2. There is clicking and distortion in the audio, which leads me to believe that my down-mixing algorithm is problematic (Yes, I am using short int's, etc.)

I am at my wit's end as to what I am doing wrong. Any help or pointers would be most appreciated.


A couple of points of clarification:

(a) Isn't linear PCM down-mixing a "simple" case of (for N channels): SampleDM = (Sample1 Sample2 ... SampleN) / N* ?

I realized that adding adding short int's together might cause an overflow before I even got the the divide-by-N part of the equation, so I distributed the division-by-N operation to each of the additions, thus:

New algorithm: Down-Mix of 1 Channel: SampleDM = (short) (Math.ceil(Sample1/N) Math.ceil(Sample2/N) Math.ceil(Sample3/N))*

(b) I'm wrapping the byte[] read from the TargetDataLine in a ShortBuffer, so I can index the samples as short int's.

(c) After manipulating (down-mixing,) For I/O to the WAV file, I am wrapping the byte[] in a ByteArrayInputStream, then passing that to a AudioInputStream, which is passed to AudioSystem.write(). Is that the best say to do this?

Call me naive or stupid if you like, please, as I am trying to learn from my msitakes.

Here is the actual code in question:

+
// Performs actual audio recording+
+public void run ()+
+{+
+// Allocate a temporary working buffer+
+int bufSize = mSession.get(0).getLine().getBufferSize(); // NB: May reduce this to 1024 or bufSize/2 to decrease latency+
+byte[] buf = new byte[bufSize];+
+mBuffer = new byte[bufSize]; // Target down-mix buffer+
+int chanCount = 0;+
+ShortBuffer sourceBuffer = ByteBuffer.wrap(buf).asShortBuffer();+
+ShortBuffer targetBuffer = ByteBuffer.wrap(mBuffer).asShortBuffer();+
+int targetIndex;+
+int sourceIndex;+
+float[] chanDivisor = new float[2];+
+chanDivisor[0] = (float)(mChannels - (mChannels / 2)); // Number of source channels down-mixed to left target channel+
+chanDivisor[1] = (float)(mChannels / 2); // Number of source channels down-mixed to right target channel+

+//System.out.println("run: bufSize="+ bufSize +" mBuffer.length="+ mBuffer.length +" mChannels="+ mChannels +" Divisors="+ chanDivisor[0] +","+ chanDivisor[1]);


// Downmix all lines' channels to one 2-channel output buffer and write to WAV file.
//
while (mRunning)
{
chanCount = 0; // Clear channel counter

// For each selected line
for (VamLine vamLine : mSession)
{
// Read audio buffer for this line (all channels, regardless of selection)
int bufLen = vamLine.getLine().read(buf, 0, bufSize);
int lineFrames = bufLen / vamLine.getLine().getFormat().getFrameSize();

// Copy from line buffer to composite buffer in correct position
for (int lineFrameIndex = 0; lineFrameIndex < lineFrames; lineFrameIndex++)
{
// Copy all channels of interest starting at first selected channel offset for selected channel count
for (int chanIndex=0; chanIndex < vamLine.getChannels(); chanIndex++)
{
if (vamLine.isSelected(chanIndex))
{
targetIndex = lineFrameIndex +(chanCount % 2); // Select left or right channel in target (round-robin)+
+sourceIndex = lineFrameIndex+ chanIndex; // Select current channel (left or right) from source
targetBuffer.put(targetIndex, (short)(targetBuffer.get(targetIndex) +(short)Math.ceil(sourceBuffer.get(sourceIndex) / chanDivisor[chanIndex])) ); // Copy channel from source to target+
+chanCount++;
}
}
}
}

// All source channels have now been down-mixed into left and right in target buffer
// Now write the target buffer to our output files.
//
//AudioInputStream ais = new AudioInputStream(bais, mAudioFormat, bufSize / 2);
ByteArrayInputStream bais = new ByteArrayInputStream(mBuffer); // Wrap buffer for AudioInputStream use
AudioInputStream ais = new AudioInputStream(bais, mAudioFormat, mBuffer.length);
for (FileOutputStream fos : mFOS)
{
try
{
ais.reset(); // Reset to beginning of buffer
AudioSystem.write(ais, AudioFileFormat.Type.WAVE, fos);
}
catch (Exception e)
{
e.printStackTrace();
}
}

// Zero-out the target buffer to get it ready for the next set of source buffers
Arrays.fill(mBuffer, (byte)0);
}
}
Thanks,
Bob

Edited by: El_Bobo on Jul 29, 2010 10:18 AM

Edited by: El_Bobo on Jul 29, 2010 10:27 AM

Edited by: El_Bobo on Jul 29, 2010 10:28 AM

Edited by: El_Bobo on Jul 29, 2010 10:36 AM

Edited by: El_Bobo on Jul 29, 2010 10:40 AM

Edited by: El_Bobo on Jul 29, 2010 10:43 AM
  • 1. Re: Multi-TargetDataLine PCM Downmixing to Single Stereo WAV File
    captfoss Pro
    Currently Being Moderated
    (a) Isn't linear PCM down-mixing a "simple" case of (for N channels): SampleDM = (Sample1 Sample2 ... SampleN) / N ?
    Yeah, that's technically correct...but I'm not sure that there's much utility to stating a general case. If you're implementing specifics, deal with specifics.

    In this instance, you'd be doing two simultanous mixing operations... two left mixes, two right mixes... so given 4 samples, you'd want to create 2 samples...
    short Left1, Left2, Right1, Right2;
    You can handle the overflow fairly easily by casting appropriately...
    short LeftOut   = (short)Math.round(((int)Left1 + (int)Left2) / 2f);
    short RightOut = (short)Math.round(((int)Right1 + (int)Right2) / 2f);
    And in the end, you'd have taken 4 samples and created 2, without any overflow issue because of the casting.
    (b) I'm wrapping the byte[] read from the TargetDataLine in a ShortBuffer, so I can index the samples as short int's.
    That would work, assuming you're not offset by a byte...then everything would go to hell, obviously.
    (c) ... I am wrapping the byte[] in a ByteArrayInputStream, then passing that to a AudioInputStream, which is passed to AudioSystem.write(). Is that the best say to do this?
    Yes.

    I have no idea what your code is doing, because it's pretty complicated. I'd dump some data from the calculations and go through by hand and make sure the calculations are being done correctly.
  • 2. Re: Multi-TargetDataLine PCM Downmixing to Single Stereo WAV File
    843802 Newbie
    Currently Being Moderated
    Hey CaptFoss,

    You rock, dude!

    I've been reading quite a few posts in this forum and your advice always seems sensible (although I wouldn't want to ever get on your bad side, if you know what I mean...) ;-)

    Anyway, a couple of things:

    1. Thanks for the feedback on the downmixing - don't know why I didn't think about promoting the short's to int's to avoid overflow. duh!

    2. I'm wondering if it's less efficient to use the ShortBuffer wrapper on my capture byte[] than to just index directly into the byte array and do bit shifting myself?

    3. I mean, I've got 4 channels to mix (2-left, 2-right) across 2 TargetDataLines and only one worker thread handle to do:

    (a) Two line.read()'s, and
    (b) Iterate through every byte in both byte[] arrays, doing the downmixing math (converting to short's, adding as rounded int's, and converting back to bytes so I can:
    (c) Store the results in the output byte[].
    (d) Pass to AudioSystem.write().

    How much byte whacking can I do on 4 channels and still stay ahead of buffer overrun on both capture lines? Guess I'll have to find out experimentally.

    4. Unanswered issue:

    (c) ... I am wrapping the byte[] in a ByteArrayInputStream, then passing that to a AudioInputStream, which is passed to AudioSystem.write(). Is that the best say to do this?

    You said "Yes" but since I am getting WAV files with incorrect "RIFF chunk" length: That is, I write 20 seconds worth of data with AudioSystem.write(), and there is 20-seconds' worth of data in the WAV file, but the header only reports either 1 or seconds' worth of data in the file.

    Then I found this thread between you and "MadMarkus": [t-5435429] "Cutting a wav file: possibly wrong datalength header in result"

    And he had the same problem as I do. Your advice to him was to use some PipedOutputStream -> PipedInputStream -> AudioInputStream.

    Something to do with the AudioSystem.write() not updating the WAV header length properly when writing to a FileOutputStream; it only works when writing to a File object.

    So, is MadMArkus McGee's problem my problem as well?

    Thanks,
    Bob

    Edited by: El_Bobo on Jul 29, 2010 2:32 PM
  • 3. Re: Multi-TargetDataLine PCM Downmixing to Single Stereo WAV File
    captfoss Pro
    Currently Being Moderated
    El_Bobo wrote:
    Hey CaptFoss,
    Hey.
    You rock, dude!
    Thanks... years of guitar practice alone in my room, I assure you...
    I've been reading quite a few posts in this forum and your advice always seems sensible (although I wouldn't want to ever get on your bad side, if you know what I mean...) ;-)
    Yes, I am a b.i.t.c.h. frequently...
    Anyway, a couple of things:

    1. Thanks for the feedback on the downmixing - don't know why I didn't think about promoting the short's to int's to avoid overflow. duh!
    My best guess as to why you didn't think to do that...
    int a = 2;
    int b = 3;
    float f = a/b;
    
    System.err.println("F = "+f);
    Will not print out 0.6667 like you'd expect... and if you do a lot (read: any) scentific kinda stuff with equations, you'll pull your hair out because of crap like that...

    In other words, been there... lol
    2. I'm wondering if it's less efficient to use the ShortBuffer wrapper on my capture byte[] than to just index directly into the byte array and do bit shifting myself?
    It's certainly less efficient to wrap an array in anything... really... but it probably isn't less efficient enough to matter.
    How much byte whacking can I do on 4 channels and still stay ahead of buffer overrun on both capture lines? Guess I'll have to find out experimentally.
    Pretty much. :-)
    4. Unanswered issue:
    So, is MadMArkus McGee's problem my problem as well?
    I don't recall the thread, but probably not...

    AudioSystem.write will either write to (1) a file (2) an output stream. If you're using it to write to a file, then you're doing it correctly. If you're writing it to some OutputStream, then you're doing it wrong...

    You may not be closing everything down correctly, though, and I'd imagine that might be where your problem is. How do you shutdown your recording?
  • 4. Re: Multi-TargetDataLine PCM Downmixing to Single Stereo WAV File
    843802 Newbie
    Currently Being Moderated
    captfoss wrote:
    El_Bobo wrote:
    Hey CaptFoss,Hey.> You rock, dude!Thanks... years of guitar practice alone in my room, I assure you...
    Yes, I recently won an air guitar competition myself, and the grand prize was a new air guitar! How cool is that? ;-)
    4. Unanswered issue:
    So, is MadMArkus McGee's problem my problem as well?I don't recall the thread, but probably not...
    I tried to paste the actual thread URL, but I guess that's a no-no, so here's the thread ID: forums.sun.com/thread.jspa?threadID=5435429

    AudioSystem.write will either write to (1) a file (2) an output stream. If you're using it to write to a file, then you're doing it correctly. If you're writing it to some OutputStream, then you're doing it wrong...
    Oops. Yes, I am using a FileOutputStream, which is derived from OutputStream, with my AudioSystem.write() invocation. My Bad.

    But the reason I did that is because I thought it was necessary to support the buffering of my byte[] manipulations. But that's not right, that would depend on the source chosen for my AudioInputStream. Hmm. Must have been late when I thought that one through. Ouch. Guess I'll fix that right now.
    You may not be closing everything down correctly, though, and I'd imagine that might be where your problem is. How do you shutdown your recording?
    Here is my stopRecording() method:
    public void stopRecording () throws IllegalStateException
    {
    if (!mRunning)
    throw new IllegalStateException("Not recording");
    
    // Stop capturing on all lines 
    for (VamLine vamLine : mSession)
    {
    vamLine.getLine().stop();
    vamLine.getLine().flush(); // Flush audio buffers
    }
    
    mRunning = false; // Stop this session
    }
    Baiscally, it does the following:

    1. Calls line().stop() and,
    2. line().flush() and,
    3. Sets my thread control variable "mRunning" to false, which causes the recording thread to exit its reading loop after the most recent AudioSystem.write() call:

    Come to think of it, I probably should probably set mRunning=false first, then do a thread join before calling the line.stop() and line.flush() methods. That way the line.read() method won't have the rug yanked out from under it during the final loop through the read/down-mix/write logic.

    I'll try these changes now and see if that clears things up.

    Thanks,
    Bob

    Edited by: El_Bobo on Jul 29, 2010 4:43 PM
  • 5. Re: Multi-TargetDataLine PCM Downmixing to Single Stereo WAV File
    captfoss Pro
    Currently Being Moderated
    El_Bobo wrote:
    Yes, I recently won an air guitar competition myself, and the grand prize was a new air guitar! How cool is that? ;-)
    I hope it was a Strat...
    I'll try these changes now and see if that clears things up.
    Letting AudioSystem.write write directly to a file will probably fix your issue, as it sounds like you do in fact have the same problem as the thread you referenced.

    You should still clean up your thread code, but, the biggest thing is how you were writing to the file. Same explanation as the previous thread.