Stanislas‑Furrer

Deep learning is limited; we can use deep learning for a continuous problem where the data is interpolative and has a learnable manifold with a dense sampling across the entire surface of the manifold between which we need to make predictions.

For Francois Chollet, generalization itself is by far the most crucial feature of artificial intelligence.

The early history of AI focused on defining priors in data structures and algorithms and tended to leave out experience and learning.

The field of AI post the deep-learning revolution seems to have the opposite obsession. In the last few years, there has been much emphasis on learning as much as possible from scratch.

The connection to Chollet’s ideas is that the deep learning era focuses on maximally exploiting experience in the form of training data.

In Chollet’s view, task-specific skills cannot tell us anything about an agent’s intelligence because it is possible to “buy arbitrary performance” by simply defining better priors or utilizing more experience. “Defining better priors or training from more experience reduces uncertainty and novelty and thus reduces the need for generalization.”

In recent years, the AI field has made massive strides in developing AI systems that learn from vast amounts of carefully labeled data. However, moving forward, it seems impossible to annotate the vast amounts of data with everything that we care about. Supervised learning is a bottleneck for allowing more intelligent generalist models to do various jobs and gain new abilities without extensive amounts of labeled data.

For Yann LeCun a promising way to approximate such common sense is self-supervised learning (SSL). Self-supervised tasks act as a proxy strategy to learn representations of the data using pseudo labels. These pseudo labels are created automatically based on the attributes found in the data. Nevertheless, the outcome of this created task is habitually dismissed. Instead, we focus on the learned intermediate representation with the hypothesis that this representation can offer excellent semantic and benefit a diversity of useful downstream tasks.

Data is everything in modern-day machine learning but is often neglected and not handled properly in AI projects. As a result, hundreds of hours are wasted on tuning a model built on low-quality data. That’s the main reason why the model's accuracy is significantly lower than expected - it has nothing to do with model tuning.

The Data-Centric Architecture treats data as a valuable and versatile asset instead of an expensive afterthought. Data-centricity significantly simplifies security, integration, portability, and analysis while delivering faster insights across the entire data value chain.

Andrew Ng’s idea is relatively simple; let us hold the model architecture fix assuming it is good enough and instead iteratively improve the data quality.

Instead of asking, “What data do I need to train a useful model?”, the question should be: “What data do I need to measure and maintain the success of my ML application?”

Modern machine learning operates with large high-quality dataset, which together with the appropriate computational resources, motivate the design of rich function space with the capacity to interpolate over the data points.

"Symmetry, as wide or narrow as you may define its meaning, is one idea by which man through the ages has tried to comprehend and create order, beauty, and perfection. (Hermann Weyl)

Since the early days, researchers have adapted neural networks to exploit the low dimensional geometric arising from physical measurements, such as grids in images, sequences in time series or position and momentum in the molecule, and their associated symmetries such as translation or translation rotation.

Remarkably, the essence of deep learning is built from two simple algorithmic principles: first, the notion of representation or feature learning, and second, learning by local gradient-descent type methods, typically implemented as backpropagation.

Geometric Deep Learning unifies a broad class of ML problems from the perspectives of symmetry and invariance. These principles not only underlie the breakthrough performance of convolutional neural networks and the recent success of graph neural networks but also provide a principled way to construct new types of problem-specific inductive biases

• The role of the quantity and quality of data in the approximation of the underlying manifold.
• The pro and cons of features engineering regarding the generalization.
• Task-specific can result in short-cut solutions that may reveal a mismatch with our intentions.

• Self-supervised learning capture the inherent structure in the data with less prior than supervised learning methods.
• The underlying concept in modern Natural language and multi-modal architecture.
• The fine-tuned solution is projected from a more general representation learn through self-supervised.

• Considering how modern Ai architecture relies on a large amount of data, it is fundamental for industries to opt for a data-centric approach.
• Managing the data drift, the data shift, the ethic and ethnic problematic, the data security and privacy, and correctly tracking and interpreting the outcome of the Ai pipeline are crucial steps for Ai large-scale deployment.

• It is fundamental to understanding the foundation and limitation of deep learning architecture