Category Archives: Mathematics

A new perspective of considering convex optimization problems based on electric circuit theory

March 6, 2025 10:41 ,

Stephen P. Boyd, Tetiana Parshakova, Ernest K. Ryu, Jaewook J. Suh, Optimization Algorithm Design via Electric Circuits, NeurIPS 2024 spotlight, 25 Sept 2024, Last Modified: 14 Jan 2025, https://openreview.net/forum?id=9Jmt1eER9P.

We present a novel methodology for convex optimization algorithm design using ideas from electric RLC circuits. Given an optimization problem, the first stage of the methodology is to design an appropriate electric circuit whose continuous-time dynamics converge to the solution of the optimization problem at hand. Then, the second stage is an automated, computer-assisted discretization of the continuous-time dynamics, yielding a provably convergent discrete-time algorithm. Our methodology recovers many classical (distributed) optimization algorithms and enables users to quickly design and explore a wide range of new algorithms with convergence guarantees.

Posted in: Mathematics , Tagged: , ,

An adaptive KF for estimating angles from IMUs

February 13, 2025 11:51 ,

Zolfa Anvari, Ali Mirhaghgoo, Yasin Salehi, Real-time angle estimation in IMU sensors: An adaptive Kalman filter approach with forgetting factor, Mechatronics, Volume 106, 2025, DOI: 10.1016/j.mechatronics.2024.103280.

In recent years, the applications of Inertial Measurement Unit (IMU) sensors have witnessed significant growth across multiple fields. However, challenges regarding angle estimation using these sensors have emerged, primarily because of the lack of accuracy in accelerometer-based dynamic motion measurements and the associated bias and error accumulation when combined with gyroscope integration. Consequently, the Kalman filter has become a popular choice for addressing these issues, as it enables the sensor to operate dynamically. Despite its widespread use, the Kalman filter requires precise noise statistics estimation for optimal noise cancellation. To accommodate this requirement, adaptive Kalman filter algorithms have been developed for estimating zero-mean Gaussian process matrix (Q) and measurement matrix (R) variances. This study introduces a real-time adaptive approach that employs a forgetting factor to precisely estimate roll and pitch angles in a 6-axis IMU. The study’s novelty lies in its algorithm, which computes the forgetting factor based on the estimation error of the last samples in the sequence. Experimental results for roll angle indicate that, in response to a step change signal, this method achieves a 54%, 39%, and 70% reduction in RMS error relative to the raw sensor data, traditional Kalman filter, and a hybrid adaptive method, respectively. Moreover, this technique exhibits significant improvements in both fixed and sinusoidal conditions for roll and pitch angles, successfully carrying out tasks within required timescales without failures related to computation time.

Posted in: Bayesian filtering , Tagged: ,

A very good explanaition of how to model certain data to further sampling from it

November 21, 2024 12:59 ,

Richard E. Turner, Cristiana-Diana Diaconu, Stratis Markou, Aliaksandra Shysheya, Andrew Y. K. Foong and Bruno Mlodozeniec, Denoising Diffusion Probabilistic Models in Six Simple Steps, arXiv:2402.04384 [cs.LG] [cs.LG].

Denoising Diffusion Probabilistic Models (DDPMs) are a very popular class of deep generative model that have been successfully applied to a diverse range of problems including image and video generation, protein and material synthesis, weather forecasting, and neural surrogates of partial differential equations. Despite their ubiquity it is hard to find an introduction to DDPMs which is simple, comprehensive, clean and clear. The compact explanations necessary in research papers are not able to elucidate all of the different design steps taken to formulate the DDPM and the rationale of the steps that are presented is often omitted to save space. Moreover, the expositions are typically presented from the variational lower bound perspective which is unnecessary and arguably harmful as it obfuscates why the method is working and suggests generalisations that do not perform well in practice. On the other hand, perspectives that take the continuous time-limit are beautiful and general, but they have a high barrier-to-entry as they require background knowledge of stochastic differential equations and probability flow. In this note, we distill down the formulation of the DDPM into six simple steps each of which comes with a clear rationale. We assume that the reader is familiar with fundamental topics in machine learning including basic probabilistic modelling, Gaussian distributions, maximum likelihood estimation, and deep learning.

Posted in: Probability and statistics , Tagged:

Fitting any dataset with a function that only has 1 parameter

September 12, 2024 11:27 ,

Laurent Bou´e, Real numbers, data science and chaos: How to fit any dataset with a single parameter, arXiv:1904.12320v1 [cs.LG] 28 Apr 2019.

We show how any dataset of any modality (time-series, images, sound…) can be approximated by a well-
behaved (continuous, differentiable…) scalar function with a single real-valued parameter. Building upon
elementary concepts from chaos theory, we adopt a pedagogical approach demonstrating how to adjust this
parameter in order to achieve arbitrary precision fit to all samples of the data. Targeting an audience of
data scientists with a taste for the curious and unusual, the results presented here expand on previous similar
observations [1] regarding expressiveness power and generalization of machine learning models.

Posted in: Mathematics , Tagged:

Procrustes analysis as a method for finding the best consensus between two sets of signals

September 12, 2024 10:58 ,

B.G.M. Vandeginste, J. Smeyers-Verbeke, Procrustes Analysis, 1998, https://www.sciencedirect.com/topics/computer-science/procrustes-analysis.

Procrustes analysis is a method in computer science that relates two sets of multivariate observations by finding the transformation that best matches the configuration of points in one set to the corresponding points in the other set, while preserving the internal structure of the objects. It involves operations such as mean centering, reflection, rotation, and finding the best match by minimizing the sum of squared distances between the transformed objects and the target configuration.

Posted in: Probability and statistics , Tagged:

Interesting review of denoising methods (applied to vision and ML, but general enough for other applications)

September 12, 2024 10:51 ,

Peyman Milanfar, Mauricio Delbracio, Denoising: A Powerful Building-Block for Imaging, Inverse Problems, and Machine Learning, arXiv:2409.06219 [cs.LG], DOI: 10.48550/arXiv.2409.06219.

Denoising, the process of reducing random fluctuations in a signal to emphasize essential patterns, has been a fundamental problem of interest since the dawn of modern scientific inquiry. Recent denoising techniques, particularly in imaging, have achieved remarkable success, nearing theoretical limits by some measures. Yet, despite tens of thousands of research papers, the wide-ranging applications of denoising beyond noise removal have not been fully recognized. This is partly due to the vast and diverse literature, making a clear overview challenging. This paper aims to address this gap. We present a comprehensive perspective on denoisers, their structure, and desired properties. We emphasize the increasing importance of denoising and showcase its evolution into an essential building block for complex tasks in imaging, inverse problems, and machine learning. Despite its long history, the community continues to uncover unexpected and groundbreaking uses for denoising, further solidifying its place as a cornerstone of scientific and engineering practice.

See also: https://en.wikipedia.org/wiki/Total_variation_denoising

Posted in: Probability and statistics , Tagged:

Predicting changes in the environment through time series for better robot navigation

September 6, 2024 07:45 ,

Yanbo Wang, Yaxian Fan, Jingchuan Wang, Weidong Chen, Long-term navigation for autonomous robots based on spatio-temporal map prediction, Robotics and Autonomous Systems, Volume 179, 2024 DOI: 10.1016/j.robot.2024.104724.

The robotics community has witnessed a growing demand for long-term navigation of autonomous robots in diverse environments, including factories, homes, offices, and public places. The core challenge in long-term navigation for autonomous robots lies in effectively adapting to varying degrees of dynamism in the environment. In this paper, we propose a long-term navigation method for autonomous robots based on spatio-temporal map prediction. The time series model is introduced to learn the changing patterns of different environmental structures or objects on multiple time scales based on the historical maps and forecast the future maps for long-term navigation. Then, an improved global path planning algorithm is performed based on the time-variant predicted cost maps. During navigation, the current observations are fused with the predicted map through a modified Bayesian filter to reduce the impact of prediction errors, and the updated map is stored for future predictions. We run simulation and conduct several weeks of experiments in multiple scenarios. The results show that our algorithm is effective and robust for long-term navigation in dynamic environments.

Posted in: Mobile robot mapping, Probability and statistics, Robot motion planning , Tagged: ,

A clustering algorithm that claims to be simpler and faster than others

June 14, 2024 14:33 ,

Yewang Chen, Yuanyuan Yang, Songwen Pei, Yi Chen, Jixiang Du, A simple rapid sample-based clustering for large-scale data, Engineering Applications of Artificial Intelligence, Volume 133, Part F, 2024 DOI: 10.1016/j.engappai.2024.108551.

Large-scale data clustering is a crucial task in addressing big data challenges. However, existing approaches often struggle to efficiently and effectively identify different types of big data, making it a significant challenge. In this paper, we propose a novel sample-based clustering algorithm, which is very simple but extremely efficient, and runs in about O(n×r) expected time, where n is the size of the dataset and r is the category number. The method is based on two key assumptions: (1) The data of each sufficient sample should have similar data distribution, as well as category distribution, to the entire data set; (2) the representative of each category in all sufficient samples conform to Gaussian distribution. It processes data in two stages, one is to classify data in each local sample independently, and the other is to globally classify data by assigning each point to the category of its nearest representative category center. The experimental results show that the proposed algorithm is effective, which outperforms other current variants of clustering algorithm.

Posted in: Machine learning , Tagged:

Using fractal interpolation for time series prediction

June 7, 2024 11:13 ,

Alexandra Băicoianu, Cristina Gabriela Gavrilă, Cristina Maria Păcurar, Victor Dan Păcurar, Fractal interpolation in the context of prediction accuracy optimization, Engineering Applications of Artificial Intelligence, Volume 133, Part D, 2024 DOI: 10.1016/j.engappai.2024.108380.

This paper focuses on the hypothesis of optimizing time series predictions using fractal interpolation techniques. In general, the accuracy of machine learning model predictions is closely related to the quality and quantitative aspects of the data used, following the principle of garbage-in, garbage-out. In order to quantitatively and qualitatively augment datasets, one of the most prevalent concerns of data scientists is to generate synthetic data, which should follow as closely as possible the actual pattern of the original data. This study proposes three different data augmentation strategies based on fractal interpolation, namely the Closest Hurst Strategy, Closest Values Strategy and Formula Strategy. To validate the strategies, we used four public datasets from the literature, as well as a private dataset obtained from meteorological records in the city of Braşov, Romania. The prediction results obtained with the LSTM model using the presented interpolation strategies showed a significant accuracy improvement compared to the raw datasets, thus providing a possible answer to practical problems in the field of remote sensing and sensor sensitivity. Moreover, our methodologies answer some optimization-related open questions for the fractal interpolation step using Optuna framework.

Posted in: Probability and statistics , Tagged: ,

Change point detection through self-supervised learning

June 7, 2024 10:53 ,

Xiangyu Bao, Liang Chen, Jingshu Zhong, Dianliang Wu, Yu Zheng, A self-supervised contrastive change point detection method for industrial time series, Engineering Applications of Artificial Intelligence, Volume 133, Part B, 2024, DOI: 10.1016/j.engappai.2024.108217.

Manufacturing process monitoring is crucial to ensure production quality. This paper formulates the detection problem of abnormal changes in the manufacturing process as the change point detection (CPD) problem for the industrial temporal data. The premise of known data property and sufficient data annotations in existing CPD methods limits their application in the complex manufacturing process. Therefore, a self-supervised and non-parametric CPD method based on temporal trend-seasonal feature decomposition and contrastive learning (CoCPD) is proposed. CoCPD aims to solve CPD problem in an online manner. By bringing the representations of time series segments with similar properties in the feature space closer, our model can sensitively distinguish the change points that do not conform to either historical data distribution or temporal continuity. The proposed CoCPD is validated by a real-world body-in-white production case and compared with 10 state-of-the-art CPD methods. Overall, CoCPD achieves promising results by Precision 70.6%, Recall 68.8%, and the mean absolute error (MAE) 8.27. With the ability to rival the best offline baselines, CoCPD outperforms online baseline methods with improvements in Precision, Recall and MAE by 14.90%, 11.93% and 43.93%, respectively. Experiment results demonstrate that CoCPD can detect abnormal changes timely and accurately.

See also: https://doi.org/10.1016/j.engappai.2024.108155

Posted in: Probability and statistics , Tagged: , , ,