Modal response-based technical countersurveillance measure against laser microphones

2.1. Instrumentation

A window is modeled by a 4 mm thick Plexiglas (acrylic glass) sheet mounted to a 500×450 mm frame with a 404×352 mm free surface area (Fig. 2). The measurement chain is schematically illustrated in Fig. 3. A SAL SBX 1320/BK medium bandwidth speaker is connected to a Genius SW-HF5.1 500 amplifier and is utilized to emulate speech in a closed room. A Keyence LK-G32 industrial laser triangulation sensor is placed at the side of the window opposite to the speaker, modeling the optical surveillance unit aimed at the window. While the construction and the working principle of this sensor are not suited for long distance or high quality intelligence gathering, it is considered here to be adequate for simulating a surveillance unit with simpler construction, respectively equipment available on the open market. The laser sensor is connected to a Keyence LK-G3001V central processing unit. The physical blocking of the optical surveillance unit is achieved by a MIDÉ QP16n lead zirconate trinitrate (PZT) transducer patch bonded onto the bottom-middle section of the acrylic glass. The transducer gains its signal from a MIDÉ EL-1225 operational amplifier.

Fig. 2Model of a windowpane under surveillance with a piezoceramic transducer mounted on its surface

Fig. 3Schematic representation of the experimental system

The measurement and algorithms are implemented on a personal computer running the Matlab xPC Target rapid control software prototyping platform, connected to hardware components with a 16 bit National Instruments PCI-6030E measurement card. The measurement system is developed in the Matlab and Simulink software environment, then loaded onto the Target platform via the TCP/IP protocol. The audio signals were sampled at 10 kHz rate with a 16 bit resolution, though not particularly high quality, but commonly used and sufficient for vocal audio [41]. The signals were later downsampled to an 8000 Hz rate. The experimental setup and a detail of the laser sensor and the piezoceramic actuator are shown in Fig. 4.

During the experiments an analog signal is supplied to the amplifier, which in turn drives the speaker emulating speech. Without the semi-active blocking signal the sound pressure causes small vibrations in the surface of the acrylic glass, which are picked up using the laser sensor emulating the signal intelligence unit. This is recorded for post-processing in the xPC Target computer. With the surveillance blocking signal activated, the investigated signal types are sent through the amplifier onto the piezoelectric transducer, disrupting the audio signature of speech inside the gathered data.

Fig. 4a) The experimental setup and b) the laser triangulation head used to emulate the laser microphone and the actuating element

a) Experimental setup

b) Laser sensor and piezoceramic transducer

2.2. Experimental identification of the resonant modes

The experimental identification of the resonant modes of the acrylic window was performed by supplying a chirp signal to the piezoceramic transducer in the bandwidth of 0-150 Hz in 150 s time with a 60 V amplitude and 1 kHz sampling. The response of the window was recorded using the laser triangulation sensor. Analyzing this response in the frequency domain, as shown in Fig. 5, indicates that the first three resonant modes are located at 31.13 Hz, 96.65 Hz and 120.90 Hz respectively. This information was used later in determining the effectiveness of a passive sinusoidal blocking signal in agreement with the first resonant frequency.

Fig. 5Periodogram power spectral density estimate of the window response from the piezoceramic transducer, showing the first resonant peak at 31.13 Hz

2.3. Speech perception quality and intelligibility evaluation

As of today, the most accurate method to compare audio speech quality is through subjective listening tests [41, 42]. The conclusions of this article will be given based on such listening tests performed by the authors, while the following sections will also contain information and conclusions based on these trials.

In addition to the formerly mentioned subjective tests, objective measures of speech quality and intelligibility were also analyzed using the algorithm collection provided by Ellis and Brookes, and are given here to allow for an unbiased evaluation of the results [43, 44]. Three different metrics are used here to quantify speech perception quality: NIST STNR which is given as the speech to noise ratio (SNR) defined as $10\mathrm{l}\mathrm{o}\mathrm{g}\left(P_{\mathrm{s}\mathrm{p}\mathrm{e}\mathrm{e}\mathrm{c}\mathrm{h},\mathrm{p}\mathrm{e}\mathrm{a}\mathrm{k}}/P_{\mathrm{n}\mathrm{o}\mathrm{i}\mathrm{s}\mathrm{e},\mathrm{m}\mathrm{e}\mathrm{a}\mathrm{n}}\right)$, where power $P$ refers to the variance of a signal computed over 20 ms windows [45]. The algorithm SNRvad gives a measure of the additional energy in speech regions determined by a crude voice activity detection (VAD) compared to the energy of the silent sections in between [43]. Finally, the perceptual evaluation of speech quality (PESQ) score is given by the fully automatized algorithm of Hu et al. [42, 46], denoting speech quality on a 1-5 scale. A method based on waveform amplitude distribution analysis (WADA) proposed by Kim and Stern in [47] was also considered in the preliminary comparison, however its results were found to be unreliable and inconsistent with the subjective perception of the signal. As the scope of this article does not allow for a deeper introduction of these methods, the reader is referred to the bibliography sources for more information.

3.1. Simulating the optical surveillance of a windowpane with post-hoc filtering

In order to allow for an easier comparison of different blocking methods, the same ~15 s long speech signal down sampled from 44 kHz audio to 10 kHz was used to simulate human conversation in a room. The audio signal used throughout all the experiments is the NASA commentary on the occasion of the landing of the space shuttle Atlantis STS-135. The transcript is: “Nose gear touchdown. Having fired the imagination of a generation, a ship like no other, its place in history secured, the space shuttle pulls into port for the last time, its voyage at an end”.

Fig. 6Normalized waveform comparison of the raw surveillance signal from the laser sensor (grey), the original speech signal (green) and the filtered surveillance signal (blue). The automatically identified active voice segments are marked with red boxes

After the laser system picked up the vibration on the surface of the window, the displacement signal was normalized to unit amplitude to increase its absolute volume [41]. Even though the resulting signal is noisy and distorted, the speech is intelligible [40]. As the real-time or post processing of the noisy signal is possible using advanced filtering techniques, the resulting surveillance data was also refined by a Wiener filter algorithm based on tracking a priori user SNR using the decision-directed method proposed by Plapous et al. [48]. It has been found that the harmonic regeneration noise reduction (HRNR) provided the best filtration results and an improvement in sound quality, however, the subjective intelligibility increase with this particular signal was not very convincing. Overall, the intelligibility of both the raw and the filtered simulated surveillance data was adequate. Albeit it was necessary to pay close attention to the reconstructed sound because of the bad quality audio, the meaning of the words and sentences could be distinguished well.

The objective speech quality metrics are given in Table 1 for the original slightly noisy speech file, the raw displacement data from the laser sensor and finally the filtered result.

Fig. 7Mean energy of the active voice segments (solid lines) and noise (dashed lines) in the original speech file (green), the raw surveillance data (grey) and the filtered surveillance data (blue)

Standard techniques used for vocal audio visualization were utilized in order to illustrate the character of the speech signals recorded during the experiments [41]. The waveform of the surveillance data is shown in Fig. 6 for the original speech without a clean reference (green), the raw displacement signal (grey) and the post-processed and filtered signal (blue). Each measurement illustrates the time series data with its maximal amplitudes normalized to unit value. Active speech regions are marked with a crude voice activity detection algorithm [43]. The mean energy for the active voice segments (solid lines) and noise segments (dashed lines) are shown in Fig. 7 for the original speech signal (green), raw unprocessed surveillance data (grey) and the filtered surveillance data (blue). The maximum-scaled sub-band energy histograms in 1 dB bins are also shown for the original speech signal in Fig. 7, raw data in Fig. 7 and post-processed data in Fig. 7.

Fig. 8Illustration of the clean speech from the original data file

a) Original signal

b) Raw surveillance

c) Filtered surveillance

The purpose of the speech perception quality indicators shown in Table 1 and the visualized speech signals in Figs. 6-8 was to give an overall assessment of the quality of the simulated surveillance signal. Unfortunately, speech perception is still mostly evaluated based on subjective testing in industrial practice [41, 42], while standard visualization techniques may not always align with the character of the real audio signal. Therefore, the indicators shown here cannot fully demonstrate the difference between speech and surveillance data. Intelligibility of speech can be objectively characterized by the PESQ score using the time domain data in Fig. 6. The PESQ score of 2.0 achieved with the filtered signal indicates a poor quality audio but intelligible speech, in agreement with examples given by McLoughlin [41] and our subjective findings. To compare the original source with simulated surveillance, it is recommended to listen to the recordings from the experiments, available online [40].

3.2. Evaluation of different semi-active blocking signals

In order to determine the best available countermeasure approach using the piezoceramic actuator, four different semi-active signal disruption strategies were tested experimentally. All methods tested in this paper use the same experimental hardware, only the types of the signals supplied to the piezoceramic transducer have been varied. We may divide the considered signal types into two main groups: random non-correlated and sinusoidal. The random signals considered here were Gaussian noise with an amplitude variance of 20 V and uniform distribution random noise with a variance of 20 V, both with zero mean. The two types of sinusoidal signals tested here were a sinusoidal signal with an arbitrarily chosen 130 Hz frequency and the other with a frequency equal to that of the first resonant mode of the acrylic window, both with ±20 V peak amplitude.

The resulting objective speech perception quality indicators are given in Table 2 for the four different countersurveillance signals. The indicator most consistent with the subjective speech quality evaluation of the authors is PESQ. Regrettably, the rest of the indicators, that is NIST [45] and SNR [43] show no clear correlation with the subjective quality and intelligibility of the audio signal.

The sinusoidal signal with the arbitrarily chosen frequency did not sufficiently degrade surveillance data quality, the filtered signal was almost identical to the one without the countersurveillance system enabled. The random signals proved to be more effective in masking the speech signal, however, still failed to completely eliminate recognizable phrases or words from the surveillance data. Of the two random approaches, the Gaussian signal was slightly better than the signal with uniform distribution. The best of the four considered methods was the harmonic signal with a frequency equal to the first resonant frequency of the window. Neither the raw unfiltered displacement data nor the filtered surveillance data contained recognizable speech.

Using mechanical and audio masking techniques to prevent laser listening, the surveillance countermeasure device may generate disturbing levels of ambient noise, which may render holding conversations difficult [1, 4]. The piezoelectric transducer combined with the window acts as a speaker generating audible noise as well. While increasing the amplitude of the masking signal and thereby increasing displacements of the window provided good results even considering the Gaussian and uniform random noise, the countersurveillance unit became increasingly noisy. The last line in Table 2 shows A-weighted sound pressure level measurements performed with a TENMA 72-860A sound level meter, which were carried out approximately ~0.2 m away from the window surface. Compared to the ambient noise level of 53.1 dB(A), the modal countersurveillance signal caused no significant increase in the noise. The perceived noise was also virtually nonexistent since the driving frequency used was well below the human hearing threshold. The rest of the evaluated methods show higher noise levels, while the sound pressure from the random masking signals proved to be very bothersome.

Naturally, the noise levels may vary depending on the material properties and geometric dimensions of the target object. The stiffness of the simulated acrylic window is less than that of glass, therefore the results on noise levels cannot be generalized. The perceived noise from the modal masking was low partly because the resonant frequency of the window was near the lower hearing threshold.

Fig. 9Illustration of the waveform of the filtered surveillance data without counter-measures (grey), and with the Gaussian (green) and modal blocking strategy (blue). Active voice segments are marked with a red line

a) Waveform comparison

b) Detail of the waveform comparison

Fig. 10Illustration of the mean energy of the active voice segments (solid lines) and noise (dashed lines) in the surveillance data without countermeasures (grey), Gaussian blocking strategy (green) and modal strategy (blue). All signals were post-processed and filtered to enhance the information content of the speech

The visualization of the surveillance data using different signals passed into the piezoceramic transducer is given in Fig. 9 and Fig. 10. The results of the experiment using a sinusoidal signal with an arbitrary frequency and normally distributed noise are omitted from these illustrations to preserve clarity. The waveform of the filtered simulated surveillance data (grey) is compared to the Gaussian noise (green) and modal sinusoidal (blue) countermeasure strategies in Fig. 9, with a detail section of the waveform presented in Fig. 8. The simulated surveillance data is compared in the frequency domain as well, Fig. 10 shows the mean energy of the active voice segments (solid lines) and noise segments (dashed line) without an active countersurveillance (grey), with the Gaussian strategy (green) and the proposed modal signal (blue). Similarly, to the visualized recordings in Section 4.1, the significance of these illustrations is limited. The time-domain waveform illustrates the degradation of the information content of the signal with the countermeasures engaged, however, for a more informative comparison the reader is referred to the recordings of the tests [40].

3.3. Properties of the modal signal blocking method

After a small reconfiguration of the experimental system and a repeated identification procedure, the modal properties of the window have been slightly altered, with the first mode now located at the somewhat higher 35.22 Hz.

The required precision of driving the piezoceramic actuator with the first resonant frequency necessary to achieve good signal blocking results was evaluated by introducing a small error in the supplied sinusoidal signal. This test evaluated subjective speech quality after performing filtering on the surveillance signal for the range of a $\pm$5 % deviation from the first resonant frequency, given in 1 % increments and decrements to both sides. For the qualitative measurements neither the NIST nor the SNRvad methods provided results showing useful correlation with subjective speech quality, therefore only the results of the PESQ algorithm by Ellis [43] (blue line) and Brookes [44] (green line) are shown in Fig. 11.

According to the subjective evaluation of the authors, a more than ±5 % deviation from the experimentally identified first resonant frequency renders the blocking signal completely inefficient and the surveillance signals start to contain recognizable words and phrases sporadically. A deviation of ±10-±20 % from the first mode renders the interference completely ineffective and the surveillance signal will be intelligible after filtering. The PESQ indicators in Fig. 11 at the extreme ends of this deviation range seem to confirm this finding.

Fig. 11Evaluation of the objective speech quality with variations in the modal frequency using different methods

Higher modes – now shifted upwards in comparison with the previous experiment – were also tested to compare their effect in disrupting the surveillance signal. While the second modal frequency located at 74.1 Hz still provided good masking of the speech signal in the surveillance data, driving the piezoceramic transducers at the third mode located at 102.3 Hz did not indicate positive results, i.e. the countersurveillance method was largely ineffective. This phenomenon is expected, since the vibration amplitude tends to decrease with higher modes, therefore the speech signal will be less disrupted by exciting the window to higher modes.