Why integrate explicit knowledge into a model, and how can we test whether it helps learning?
01.01 / Understand
The starting point: exoplanets and ExoMiner
My starting point was exoplanets. While working with Kepler and TESS data and discovering ExoMiner, the NASA project integrated into my application, an intuition took shape: why ask a network to rediscover on its own what we already know about the phenomenon it observes?
In my work on light curves, the signal was not the only information available. The properties of the star and the physical information could also guide the interpretation, notably through the tokens [STAR] and [PHYS]. A drop in brightness takes on another meaning when put into context. This precedent inspires the thesis; it does not yet demonstrate that the same approach will work for action.
The passage of a planet in front of its star reduces the light received. Schematic curve, without Kepler or TESS measurements.
01.02 / Connect
The transmission of knowledge as a working hypothesis
People do not experience every situation from scratch. We exchange explanations, rules and experiences. This knowledge broadens what we can consider; we then make choices in the situation we encounter. Shared information can save many unnecessary trials.
But we can also receive bad advice. The loop I am interested in is therefore more complete: receive knowledge, imagine its implications, try it out, compare it with what happens and revise our trust in it. The project seeks a computational expression of this experimental loop, not a complete reproduction of human cognition.
Principle diagram / Understanding the mechanismThe human analogy guiding the project. Automatically revising the knowledge base is a future direction, not an already demonstrated capability.
01.03 / Put to the test
From classification to prediction of the effects of an action
For exoplanets, the question was: what does this signal mean, given what I know? For control, it becomes: what will happen if I perform this action, given what I know about the world? An agent must predict consequences before choosing.
Three hypotheses make this intuition testable. H1: correct knowledge reduces the number of interactions needed. H2: reusable rules help with unseen combinations. H3: learning to regulate their influence limits damage when they are incomplete or false. Each will need to be measured separately.
The target changes: interpreting a signal, then anticipating the position of an object after an action.
It's your turn to exploreEducational model
What are we looking to check?
Choose a hypothesis to discover the experiment that could test it.
Can useful knowledge win essays?
Compare with and without knowledge, then measure the number of transitions necessary to reach the same error threshold.
Three experimental questions; their answers remain to be established.
Sources for this chapter01 references +
Project / Reference of the exoplanets projectExoMiner — NASA ↗NASA Ames · ExoMiner team
Why this sourceStarting point: analysis of Kepler and TESS transits. To be linked to the exoplanets project and the question of the use of physical information.
Framework document, p. 2–3, 37. The hypotheses and protocols presented remain to be tested.
Which studies already address this problem, and how can we define the proposed contribution?
02.01 / Understand
Organising the literature review
I group the sources according to their role in the project. First understand how to predict in a latent space. Then understand why objects, their relationships and possible actions can help. Then compare the idea to the closest approaches, before considering larger visual models.
Adding knowledge to an agent is not new in itself. The specific question here concerns explicit, editable knowledge: its influence on dynamics, the cost of interactions and the trust placed in it when it is wrong. The bibliography should help define that contribution, without claiming that no other work covers it.
Principle diagram / Understanding the mechanismA conceptual reading order: understand, structure, compare, then extend. The reading statuses remain those recorded in the journal.
02.02 / Connect
Criteria for analyzing publications
A useful reference not only provides an encouraging result. I want to identify its hypothesis, the knowledge actually accessible to the model, the comparison protocol and the limits of the transfer tested. A rule learned from examples does not have the same role as an external base that can be modified by hand.
The notes below explain why each source matters and at which stage it is useful. They bring together the work cited in the scoping document and the references recorded in the journal. A listed source has not necessarily been read in full.
For each source: identify the hypothesis, the accessible information, the baselines and the transfer situations tested.
Why this sourceTrack the link between a frozen world model and an affordance prior used for manipulation; clarify the boundary of the proposed contribution.
Why this sourceFollow the convergence of vision, language, action and affordances. Used to locate future directions, not to impose a large model on the first experiment.
05 / Initial project and evaluation environments4 ref.
Steps 01 & 06 · Exoplanets and environments
Project / Reference of the exoplanets projectExoMiner — NASA ↗NASA Ames · ExoMiner team
Why this sourceStarting point: analysis of Kepler and TESS transits. To be linked to the exoplanets project and the question of the use of physical information.
Why this sourceOther terrain cited in the document to reflect on the transition to visual observations and driving; to compare to the kinematic prototype.
Sources for this chapter00 references +
Framework document, p. 3–7, 14–16, 37–38. The hypotheses and protocols presented remain to be tested.
Which representation makes it possible to predict transitions useful for control?
03.01 / Understand
The role of the encoder and the latent state
The observation o contains what the agent receives: positions, speeds, or later an image. An encoder E transforms it into latent state z, a compact internal representation. The word latent simply designates information constructed by the model, which is not directly observed.
This compression must preserve what makes it possible to predict the effects of actions. If I push an object, its position, mobility and obstacles matter. The decorative texture of the wall can be secondary. What is useful, however, depends on the problem: you cannot decide in advance that all visual details are useless.
The encoder represents the scene by variables useful for prediction. The drawn components are symbolic.
03.02 / Connect
Learning transitions and multi-step predictions
Dynamics model F takes the current latent state and an action a, then predicts the next state. During training, this prediction is compared with the representation of what was actually observed. By repeating this process, the model learns regularities in how things change.
We can then chain together several predictions: a rollout is an imagined trajectory. But a small mistake can propagate with every step. It is therefore necessary to evaluate forecasts over several horizons, and not just the next transition. PlaNet and TD-MPC2 are benchmarks for understanding this transition from prediction to planning.
Principle diagram / Understanding the mechanismo: observation; E: encoder; z: latent state; a: action; F: dynamics. The training target comes from the actual next observation.
03.03 / Put to the test
The choice of a first prototype with vector states
The first prototype can directly receive a state vector. This makes it possible to study the dynamics without confusing a perception error with an error in reasoning about transitions. There is no need to start with videos, a large model, or a physical robot.
Later, a frozen visual encoder could replace this input adapter. DINO-WM and V-JEPA 2 inform that direction. The central question will remain the same: how does knowledge change a prediction, and when should its influence be reduced?
Providing the state variables directly allows you to study the dynamics before adding a visual encoder.
How to represent K and isolate its influence on the prediction of the next state?
04.01 / Understand
The representation of affordances and preconditions
An affordance is an opportunity for action in a situation: an object can be grasped, a drawer opened, a surface crossed. It depends on context and the agent’s capabilities. Saying that a cup can be grasped does not mean it can be grasped in every position or with any gripper.
Knowledge base K brings together entities, relationships, possible actions and constraints. For example: the handle belongs to the drawer; pulling the drawer requires grasping its handle; placing a cup inside requires the drawer to be open. This small language makes the assumptions visible.
Principle diagram / Understanding the mechanismAn example of preconditions. The graph gives a usable structure; it does not provide future coordinates or the optimal trajectory.
04.02 / Connect
The addition of a residual correction conditioned by K
The base model F learns from observations. A second branch G proposes a correction derived from relevant knowledge. The final prediction combines them with weight α. This separation makes the contribution of knowledge easier to test: remove the correction, replace K or fix its weight.
A readable and editable database also provides a means of intervention. If we remove a relationship or invert a property, we can observe what changes in the prediction. This experiment is not enough to prove causal understanding, but it does verify that the model actually uses this information.
Proposed architecture: F predicts the base transition; G provides a correction conditioned by K, weighted by α.
04.03 / Put to the test
Controlling the information provided to the model
The comparison would be misleading if the knowledge base directly revealed the future state, the goal or a solution unavailable to the baseline. We must document exactly what K contains, how it is obtained and what it costs to build.
We must also compare to simpler solutions: concatenating attributes, adding a task identifier or filtering impossible actions. If a mask of actions is enough to obtain the same gain, a complex graph is not yet justified.
Relations and preconditions are documented. A future response or optimal solution would constitute privileged information.
How can we reduce the influence of erroneous knowledge and assess the reliability of predictions?
05.01 / Understand
The role of the weighting coefficient α
Weight α, between 0 and 1, modulates correction G. Near zero, it reduces its influence; near one, it allows it to contribute fully. The weight must depend on the state, action and knowledge involved. A rule that is useful here may be unsuitable elsewhere.
The aim is to learn this balance from experience. If K claims that a wall can be grasped, observed transitions should allow the model to challenge that advice. But poor knowledge can also contaminate learning: a trust gate alone does not guarantee a safe return to the model without knowledge.
Principle diagram / F + αGProposed architecture: basic prediction + weighted correction. α regulates an influence; it is not automatically a probability of truth.
05.02 / Connect
The distinction between weighting and calibrated uncertainty
Weight α answers “how much influence should this information have?” Uncertainty u answers “how reliable is my prediction?” These notions are not interchangeable. A model can ignore K and remain very uncertain because it also lacks experience.
Calibrating an uncertainty means comparing what it announces with the errors actually observed on separate data. For example, a predicted 90% prediction interval should cover approximately 90% of the targets in the evaluated context. We must also look at its width: immense intervals would cover everything without helping to take action.
α modulates the correction. The intervals represent uncertainty: their coverage and width must be evaluated separately. Illustration without results.
05.03 / Put to the test
Knowledge Base Corruption Testing
The protocol will vary K’s quality: correct, partial, absent and contradictory. Removing relationships measures the effect of missing information; introducing false ones poses a different difficulty. These two perturbations must therefore remain distinct.
We will measure prediction error, degradation against the baseline without K, the behaviour of α and calibration over several horizons. A low weight on a false knowledge base is a useful indicator, but the evidence we seek is the system’s actual behaviour in situations held out from tuning.
A missing relationship and a false relationship are two distinct disturbances. Their impact must be measured against the control without K.
It's your turn to exploreEducational model
What place should be given to advice?
The model predicts 4. The target is 6. Vary the weight of the correction and change the quality of the advice.
Basis · 4Target · 6↑08
4 + 0.50 × (+2) = 5.00Deviation from target: 1.00
Here you fix α by hand. The project seeks to learn this. This weight is not a calibrated measure of uncertainty.
At a fixed prediction accuracy, how many interactions does integrating K save?
06.01 / Understand
Comparing models with equal data budget
The first experiment uses transitions: a state, an action, then the state actually obtained. For the same observation budget, we compare a model without knowledge to a model that can use K. The same data divisions and the same random seeds must be preserved between methods.
We vary the number of training transitions. Sample efficiency is measured by the budget needed to reach a predefined error threshold, and by error at an equal budget. Computation time, parameter count and the cost of K complete the picture: saving interactions does not automatically save every resource.
Principle diagram / Understanding the mechanismSeparate training, validation and test data before running experiments. The error threshold is a criterion to set, not a result already achieved.
06.02 / Connect
Reference models and ablations
The planned baselines are: dynamics alone, task conditioning, concatenated attributes, an encoded graph, an affordance mask, a weighted correction and possibly a symbolic oracle. The oracle is a privileged reference; its additional information must be disclosed.
An ablation removes part of the method to measure its contribution. Without G, we test the correction. Without α, we test its weighting. With unrelated attributes, we test the value of structure. Comparable capacities and budgets prevent us from confusing the contribution of knowledge with a simple increase in network size.
This matrix describes three configurations, without scores: F alone, F + G with fixed coefficient, then F + αG with learned coefficient.
06.03 / Put to the test
Reproducibility and limits of the current prototype
Each trial must retain its configuration, code version, budget, seeds, data splits, data or provenance, and predictions. Averages must come with measures of variability; failures deserve as much space as successes.
The lab in the notes below already illustrates scalar dynamics with a correction and a weight fitted on validation data. It does not validate the thesis: the latent state is not learned, K is synthetic and privileged, and transfer is not compositional. The thesis experiments still need to be carried out and documented.
Linking predictions to code and data versions, budget and seed allows you to find the conditions of each test.
Does the model correctly predict object–relation–action compositions absent from training?
07.01 / Understand
The construction of compositional generalisation partitions
Compositional generalisation consists of succeeding on a never before seen combination of known elements. The model may have learned to push a ball and place a cube in a box; We then reserve for the test the placement of a ball in a container. Knowing each component does not guarantee mastery of this new combination.
The training–test split must be made by composition before producing transitions. A simple random shuffle risks putting the same combination on both sides. New objects, new properties and new relationships present different challenges: results must distinguish them.
Principle diagram / Understanding the mechanismExample of composition reserved for the test. No transitions in this combination should be used for training or adjustment.
07.02 / Connect
From prediction to receding-horizon control
Once the dynamics have been validated, the agent could imagine several action sequences, compare their usefulness, constraints and uncertainty, then execute the first selected action. A new observation would allow it to correct the prediction and start again: this is the principle of receding-horizon control, or MPC.
The proposed progression starts with a synthetic world, moves to an inexpensive control environment, then to a small family of robotic tasks. HighwayEnv and Meta-World are directions cited in the document. Moving between domains will be another experiment, not generalisation we can take for granted.
The dotted lines represent the imagined sequences. Only the first chosen action is executed, then the agent observes and recalculates.
07.03 / Put to the test
Extensions conditioned by experimental results
What happens next will depend on the results. A gain may come from simple action filtering; a failure may reveal knowledge poorly connected to observations, an unresponsive trust gate or an overly favourable protocol. These explanations should guide the next experiment.
In the longer term, visual perception, automatic knowledge revision and transfer between domains would broaden the programme. The aim remains the same: draw inspiration from how we receive knowledge, test it and question it to learn more effectively.
The results should determine whether it is necessary to review the method or undertake new evaluations in vision, control and transfer.
It's your turn to exploreEducational model
Known separately. Unpublished together.
In training: push a ball and place a cube. Compose an action to distinguish storage and transfer.
+
Combination reserved for testingPlace a ball in a box.
The action and the object are known, their association is new. It is this difference that the test focuses on.
This manipulation explains the division of the data; it does not simulate a trained model.
Why this sourceOther terrain cited in the document to reflect on the transition to visual observations and driving; to compare to the kinematic prototype.
Framework document, p. 17–18, 20–23, 35–37. The hypotheses and protocols presented remain to be tested.