# Thousand Brains Project Documentation

> Documentation for Thousand Brains Project

## Guides
- [Welcome to the Thousand Brains Project Documentation!](https://docs.thousandbrains.org/docs/welcome-to-the-thousand-brains-project-documentation.md)
- [Vision of the Thousand Brains Project](https://docs.thousandbrains.org/docs/vision-of-the-thousand-brains-project.md)
- [Long-Term Goals and Principles](https://docs.thousandbrains.org/docs/long-term-goals-and-principles.md)
- [Short-Term Goals](https://docs.thousandbrains.org/docs/short-term-goals.md)
- [Challenging Preconceptions](https://docs.thousandbrains.org/docs/challenging-preconceptions.md)
- [Capabilities of the System](https://docs.thousandbrains.org/docs/capabilities-of-the-system.md)
- [Architecture Overview](https://docs.thousandbrains.org/docs/architecture-overview.md)
- [Sensor Modules](https://docs.thousandbrains.org/docs/sensor-modules.md)
- [Learning Modules](https://docs.thousandbrains.org/docs/learning-modules.md)
- [Cortical Messaging Protocol](https://docs.thousandbrains.org/docs/cortical-messaging-protocol.md)
- [Other Aspects](https://docs.thousandbrains.org/docs/other-aspects.md)
- [Bringing it Together](https://docs.thousandbrains.org/docs/bringing-it-together.md)
- [Application Criteria](https://docs.thousandbrains.org/docs/application-criteria.md): What can you use Monty for?
- [Benchmark Experiments](https://docs.thousandbrains.org/docs/benchmark-experiments.md): Performance of current implementation on our benchmark test suite.
- [Results from Alternative Implementations](https://docs.thousandbrains.org/docs/results-from-alternative-implementations.md)
- [FAQ - Thousand Brains Project](https://docs.thousandbrains.org/docs/faq-thousand-brains-project.md): Frequently asked question about the project.
- [Glossary](https://docs.thousandbrains.org/docs/glossary.md): This section aims to provide concise definitions of terms commonly used at the Thousand Brains Project and in Monty.
- [Further Reading](https://docs.thousandbrains.org/docs/further-reading.md): Here we put a list of books and papers that might be interesting for you if you are interested in this project.
- [Community and Media Coverage](https://docs.thousandbrains.org/docs/community-and-media-coverage.md)
- [Getting Started](https://docs.thousandbrains.org/docs/getting-started.md): How to get the code running.
- [Getting Started on Windows via WSL](https://docs.thousandbrains.org/docs/getting-started-on-windows-via-wsl.md): How to get the code running using Windows Subsystem for Linux and Visual Studio Code
- [Code Base Structure](https://docs.thousandbrains.org/docs/code-base-structure.md)
- [Running Benchmarks](https://docs.thousandbrains.org/docs/running-benchmarks.md): Benchmark experiments should be run with every functional change to the code. This is how you do it.
- [Logging and Analysis](https://docs.thousandbrains.org/docs/logging-and-analysis.md)
- [Customizing Monty](https://docs.thousandbrains.org/docs/customizing-monty.md)
- [Implementing Actions](https://docs.thousandbrains.org/docs/implementing-actions.md)
- [Tutorials](https://docs.thousandbrains.org/docs/tutorials.md)
- [Running Your First Experiment](https://docs.thousandbrains.org/docs/running-your-first-experiment.md)
- [Pretraining a Model](https://docs.thousandbrains.org/docs/pretraining-a-model.md)
- [Running Inference with a Pretrained Model](https://docs.thousandbrains.org/docs/running-inference-with-a-pretrained-model.md)
- [Unsupervised Continual Learning](https://docs.thousandbrains.org/docs/unsupervised-continual-learning.md)
- [Multiple Learning Modules](https://docs.thousandbrains.org/docs/multiple-learning-modules.md)
- [Using Monty in a Custom Application](https://docs.thousandbrains.org/docs/using-monty-in-a-custom-application.md)
- [Using Monty for Robotics](https://docs.thousandbrains.org/docs/using-monty-for-robotics.md)
- [Running An Experiment From A Different Repository](https://docs.thousandbrains.org/docs/running-an-experiment-from-a-different-repository.md)
- [Common Issues (And how to Fix Them)](https://docs.thousandbrains.org/docs/common-issues-and-how-to-fix-them.md)
- [Implementation Overview](https://docs.thousandbrains.org/docs/implementation-overview.md)
- [Experiment](https://docs.thousandbrains.org/docs/experiment.md)
- [Environment & Agent](https://docs.thousandbrains.org/docs/environment-agent.md)
- [Monty](https://docs.thousandbrains.org/docs/monty.md)
- [Messages in Monty](https://docs.thousandbrains.org/docs/messages-in-monty.md)
- [Reference Frame Transformations](https://docs.thousandbrains.org/docs/reference-frame-transformations.md)
- [Conventions](https://docs.thousandbrains.org/docs/conventions.md)
- [Sensor Module](https://docs.thousandbrains.org/docs/sensor-module.md)
- [SM for RGBD data](https://docs.thousandbrains.org/docs/sm-for-rgbd-data.md)
- [SalienceSM](https://docs.thousandbrains.org/docs/salience-sm.md)
- [TwoDSensorModule](https://docs.thousandbrains.org/docs/two-d-sensor-module.md)
- [Learning Module](https://docs.thousandbrains.org/docs/learning-module.md)
- [Object Models](https://docs.thousandbrains.org/docs/object-models.md)
- [Learning Module Outputs](https://docs.thousandbrains.org/docs/learning-module-outputs.md)
- [Evidence Based Learning Module](https://docs.thousandbrains.org/docs/evidence-based-learning-module.md)
- [Connecting LMs into a Heterarchy](https://docs.thousandbrains.org/docs/connecting-lms-into-a-heterarchy.md)
- [Motor System](https://docs.thousandbrains.org/docs/motor-system.md)
- [Policy](https://docs.thousandbrains.org/docs/policy.md)
- [FAQ - Monty](https://docs.thousandbrains.org/docs/faq-monty.md): Frequently asked questions about the Thousand Brains Theory, Monty, thousand-brains systems, and the underlying algorithms.
- [Open Questions](https://docs.thousandbrains.org/docs/open-questions.md): Below is a simple outline of some of the open questions that we are currently exploring.
- [Why Contribute?](https://docs.thousandbrains.org/docs/why-contribute.md)
- [Ways to Contribute](https://docs.thousandbrains.org/docs/contributing.md)
- [Pull Requests](https://docs.thousandbrains.org/docs/pull-requests.md)
- [Contributor License Agreement](https://docs.thousandbrains.org/docs/contributor-license-agreement.md)
- [Pull Request Flow](https://docs.thousandbrains.org/docs/pull-request-flow.md)
- [Reviewing a Pull Request](https://docs.thousandbrains.org/docs/reviewing-a-pull-request.md)
- [Ways to Contribute to Code](https://docs.thousandbrains.org/docs/ways-to-contribute-to-code.md)
- [Identify an issue to work on](https://docs.thousandbrains.org/docs/identify-an-issue-to-work-on.md)
- [Documentation](https://docs.thousandbrains.org/docs/documentation.md)
- [Future Work Widget Metadata](https://docs.thousandbrains.org/docs/future-work-widget-metadata.md)
- [Contributing Tutorials](https://docs.thousandbrains.org/docs/contributing-tutorials.md)
- [Comment on Issues](https://docs.thousandbrains.org/docs/comment-on-issues.md)
- [Request For Comments (RFC)](https://docs.thousandbrains.org/docs/request-for-comments-rfc.md)
- [Style Guide](https://docs.thousandbrains.org/docs/style-guide.md)
- [Code Style Guide](https://docs.thousandbrains.org/docs/code-style-guide.md)
- [Typing Guide](https://docs.thousandbrains.org/docs/typing-guide.md)
- [Guides for Maintainers](https://docs.thousandbrains.org/docs/guides-for-maintainers.md)
- [Triage](https://docs.thousandbrains.org/docs/triage.md)
- [Large Language Model Use](https://docs.thousandbrains.org/docs/large-language-model-use.md)
- [Code of Conduct](https://docs.thousandbrains.org/docs/code-of-conduct.md)
- [Governance](https://docs.thousandbrains.org/docs/governance.md)
- [Our Licence and Patent Pledge Explained](https://docs.thousandbrains.org/docs/our-licence-and-patent-pledge-explained.md)
- [TBP Based Papers](https://docs.thousandbrains.org/docs/tbp-based-papers.md): A list of papers that are based on the TBP or the TBT.
- [Project Showcase](https://docs.thousandbrains.org/docs/project-showcase.md): A list of projects that utilize Monty or the TBT.
- [Project Roadmap](https://docs.thousandbrains.org/docs/project-roadmap.md): An overview of tasks we plan to work on or would welcome contributions on.
- [Sensor Module Improvements](https://docs.thousandbrains.org/docs/sensor-module-improvements.md): Improvements we would like to add to the sensor modules.
- [Extract Better Features](https://docs.thousandbrains.org/docs/extract-better-features.md): Add more sophisticated feature extraction to sensor modules.
- [One-D Sensor Module](https://docs.thousandbrains.org/docs/one-d-sensor-module.md): Sensor module that converts 3D movement in space to 1D movement along a principal direction.
- [Parallax for Depth Perception](https://docs.thousandbrains.org/docs/parallax-for-depth-perception.md): Implement a sensor module that can estimate depth based on binocular or motion parallax, reducing the need for a depth sensor.
- [Detect Local and Global Flow](https://docs.thousandbrains.org/docs/detect-local-and-global-flow.md): Add capability to SM to detect flow (direction & velocity) within its receptive field.
- [Change Detecting SM](https://docs.thousandbrains.org/docs/change-detecting-sm.md): Add a new type of sensor module that detects local changes and output's those as CMP messages.
- [Learning Module Improvements](https://docs.thousandbrains.org/docs/learning-module-improvements.md): Improvements we would like to add to the learning modules.
- [Contributing Learning Modules](https://docs.thousandbrains.org/docs/contributing-learning-modules.md): Test different algorithms or representations inside of LMs.
- [Use Off-Object Observations](https://docs.thousandbrains.org/docs/use-off-object-observations.md): Ensure that off-object observations are processed by LMs, resulting in evidence updates.
- [Use Out of Model Movements](https://docs.thousandbrains.org/docs/use-out-of-model-movements.md): Process out-of-model-movements such that we can distinguish similar objects, while also eliminating poor hypotheses.
- [Improve Bounded Evidence Performance](https://docs.thousandbrains.org/docs/improve-bounded-evidence-performance.md): If we don't allow evidence to grow infinitely, memory fades. Develop mechanisms to deal with this.
- [Use Models with Fewer Points](https://docs.thousandbrains.org/docs/use-models-with-fewer-points.md): Update learning algorithm, sensor module, and policy to learn sparser models.
- [Better Model Surface Representation](https://docs.thousandbrains.org/docs/better-model-surface-representation.md): Figure out and test a way to represent an object's surface better in our models and/or matching algorithm.
- [Hybrid Node and Edge Matching](https://docs.thousandbrains.org/docs/hybrid-node-and-edge-matching.md): Combine the matching approach taken in the EvidenceGraphLM (nodes) with the one in the DisplacementGraphLM (edges).
- [Sample Offspring Hypotheses](https://docs.thousandbrains.org/docs/sample-offspring-hypotheses.md): Sample new hypotheses that are similar to the existing top hypotheses.
- [Re-Anchor Hypotheses for Robustness to Noise and Distortions](https://docs.thousandbrains.org/docs/re-anchor-hypotheses.md): Reanchor hypothesized poses to nodes stored in model when salient features with high match are observed.
- [Deal with Incomplete Models](https://docs.thousandbrains.org/docs/deal-with-incomplete-models.md): Monty has no concept of whether an object doesn't exist at a location or it's model is just incomplete.
- [Test Grid Object Models for Unsupervised Learning](https://docs.thousandbrains.org/docs/test-grid-object-models-for-unsupervised-learning.md): Evaluate and finetune the GridObjectModels in unsupervised settings.
- [Unsupervised Learning in a Hierarchy](https://docs.thousandbrains.org/docs/unsupervised-learning-in-a-hierarchy.md): Test and improve learning with multiple LMs stacked on top of each other without supervising them.
- [Learning and Forgetting Speed Parameter](https://docs.thousandbrains.org/docs/learning-and-forgetting-speed-parameter.md): Add a learning speed parameter to object models in LMs such that some LMs learn more temporary models and others more stable ones.
- [Deal with Moving Objects](https://docs.thousandbrains.org/docs/deal-with-moving-objects.md): Use sensed local flow or predicted object movement to compensate the hypothesized location on object.
- [Support Scale Invariance](https://docs.thousandbrains.org/docs/support-scale-invariance.md): Add a mechanism to detect and compensate for scale of an object.
- [Improve Handling of Symmetry](https://docs.thousandbrains.org/docs/improve-handling-of-symmetry.md): Remember symmetric poses to recognize symmetry faster on subsequent exposures.
- [Model Repeating Patterns](https://docs.thousandbrains.org/docs/model-repeating-patterns.md): Come up with, implement, and test a way to model and recognize repeating patterns efficiently.
- [Use Better Priors for Hypothesis Initialization](https://docs.thousandbrains.org/docs/use-better-priors-for-hypothesis-initialization.md): Inform initial hypotheses by common orientations an object is observed in and context.
- [Learn Hot Spots on Objects](https://docs.thousandbrains.org/docs/learn-hot-spots-on-objects.md): Store locations on objects that helped most in recognizing it or that contain particularly salient features to use as heuristics when attending.
- [Include State in Models](https://docs.thousandbrains.org/docs/include-state-in-models.md): Add a 'state' dimension to the models learned in Monty that conditions which features to expect at what locations.
- [Include State in Hypotheses](https://docs.thousandbrains.org/docs/include-state-in-hypotheses.md): Infer state of object, in addition to its pose.
- [Event Detection to Reset Timer](https://docs.thousandbrains.org/docs/event-detection-to-reset-timer.md): Add ability to detect significant events which are used as signals to reset the global interval timer.
- [Speed Detection to Adjust Timer](https://docs.thousandbrains.org/docs/speed-detection-to-adjust-timer.md): Add ability to detect offsets in timer input and learned sequence model to speed up or slow down the global interval timer.
- [Motor System Improvements](https://docs.thousandbrains.org/docs/motor-system-improvements.md): Improvements we would like to add to the motor system.
- [Implement Switching Between Learning and Inference-Focused Policies](https://docs.thousandbrains.org/docs/implement-switching-between-learning-and-inference-focused-policies.md): Dynamically switch the way goals are selected based on more high level goals and confidence of the system.
- [Bottom-Up Exploration Policy for Surface Agent](https://docs.thousandbrains.org/docs/bottom-up-exploration-policy-for-surface-agent.md): A naive scan policy equivalent for the surface agent.
- [Model-Based Exploration Policy](https://docs.thousandbrains.org/docs/model-based-exploration-policy.md): Use LM models during exploration to move to most uncertain areas in model and learn more about those.
- [Learn Policy using RL and Simplified Action Space](https://docs.thousandbrains.org/docs/learn-policy-using-rl.md): Test using reinforcement learning to learn policies.
- [Theory for Decomposing Goals into Subgoals](https://docs.thousandbrains.org/docs/theory-for-decomposing-goals-into-subgoals.md): Figure out how we can decompose higher-level goals into sub-goals and communicate them
- [Policy to Recognize an Object Before Moving onto a New Object](https://docs.thousandbrains.org/docs/policy-to-recognize-an-object-before-moving-on-to-a-new-object.md): Use the SM and/or LM goal generator to remain on an object until it is recognized (e.g. using basic image segmentation)
- [Policy to quickly move to a new object](https://docs.thousandbrains.org/docs/policy-to-quickly-move-to-a-new-object.md): Once an object is recognized, use its learned model to move off the object.
- [Voting Improvements](https://docs.thousandbrains.org/docs/voting-improvements.md): Improvements we would like to add to voting.
- [Clean up and Simplify Voting](https://docs.thousandbrains.org/docs/clean-up-and-simplify-voting.md): Simplify some of the voting code to make it more efficient and easier to understand.
- [Outline How to Apply Attention](https://docs.thousandbrains.org/docs/outline-how-to-apply-attention.md): Outline how we can gate information in Monty's routing (votes and inputs) to apply attention filters to regions of space.
- [Generalize Voting to Associative Connections](https://docs.thousandbrains.org/docs/generalize-voting-to-associative-connections.md): Learn associations between object representations in different LMs instead of requiring them to use the same representation for object ID.
- [Can we change the CMP to use displacements instead of locations?](https://docs.thousandbrains.org/docs/can-we-change-the-cmp-to-use-displacements-instead-of-locations.md): Investigate using movement vectors instead of locations in the CMP and the implications it would have for Monty.
- [Vote on State](https://docs.thousandbrains.org/docs/vote-on-state.md): Update the voting algorithm to take the state of an object into account.
- [CMP & Hierarchy Improvements](https://docs.thousandbrains.org/docs/cmp-hierarchy-improvements.md): Improvements we would like to add to the CMP or hierarchical information processing.
- [Add Top-Down Connections](https://docs.thousandbrains.org/docs/add-top-down-connections.md): Add top-down connectivity option between LMs that can inform the receiving LM of the object ID and location it should likely be sensing.
- [Run & Analyze Experiments with more LMs in Heterarchy Testbed](https://docs.thousandbrains.org/docs/run-analyze-experiments-with-more-lms-in-heterarchy-testbed.md): Scale our current hierarchy experiments to more LMs and test the impact of this.
- [Evaluate on Omniglot Dataset](https://docs.thousandbrains.org/docs/evaluate-on-omniglot-dataset.md): Test the ability to model compositional objects on the Omniglot dataset.
- [Test Learning at Different Speeds Depending on Level in Hierarchy](https://docs.thousandbrains.org/docs/test-learning-at-different-speeds-depending-on-level-in-hierarchy.md): Modify constrained object models to learn at different speeds and test the effect of slower learning at lower levels in the hierarchy.
- [Send Similarity Encoding Object ID to Next Level & Test](https://docs.thousandbrains.org/docs/send-similarity-encoding-object-id-to-next-level-test.md): Test the EvidenceSDRLMMixin in a hierarchical Monty setup to model compositional objects.
- [Global Interval Timer](https://docs.thousandbrains.org/docs/global-interval-timer.md): Add a (semi) global interval timer that provides information about time elapsed since the last event to LMs.
- [Include State in CMP](https://docs.thousandbrains.org/docs/include-state-in-cmp.md): Include the inferred state of an object as part of the CMP message.
- [Environment Improvements](https://docs.thousandbrains.org/docs/environment-improvements.md): New environments and benchmark experiments we would like to add.
- [Set Up Replicable Real World Test Bed](https://docs.thousandbrains.org/docs/replicable-real-world-test-bed.md): Create a real-world test bed for Monty with instructions on how to replicate it.
- [Object Behavior Test Bed](https://docs.thousandbrains.org/docs/object-behavior-test-bed.md): Set up an environment to test modeling and inferring object behavior under various conditions.
- [Make Dataset to Test Compositional Objects](https://docs.thousandbrains.org/docs/make-dataset-to-test-compositional-objects.md): Create more complex compositional object meshes that we can load in our simulator and test Monty on.
- [Set up Environment that Allows for Object Manipulation](https://docs.thousandbrains.org/docs/set-up-environment-that-allows-for-object-manipulation.md): Identify a simple setup for simulating and manipulating objects and turn it into an environment for Monty.
- [Set Up Object Manipulation Benchmark Tasks and Evaluation Measures](https://docs.thousandbrains.org/docs/set-up-object-manipulation-benchmark-tasks-and-evaluation-measures.md)
- [Create Dataset and Metrics to Evaluate Categories and Generalization](https://docs.thousandbrains.org/docs/create-dataset-and-metrics-to-evaluate-categories-and-generalization.md)
- [Create Dataset to Test new Feature-Morphology Pairs](https://docs.thousandbrains.org/docs/create-dataset-to-test-new-feature-morphology-pairs.md): Create a mesh dataset where we can test learning specific object-texture pairs and then infer arbitrary combinations.
- [Framework Improvements](https://docs.thousandbrains.org/docs/framework-improvements.md): Improvements we would like to make on the general code framework.
- [Add Infrastructure for Multiple Agents that Move Independently](https://docs.thousandbrains.org/docs/add-infrastructure-for-multiple-agents-that-move-independently.md): Currently, Monty only supports one Agent. Update Monty's setup and motor system to support multiple agents.
- [Automate Benchmark Experiments & Analysis](https://docs.thousandbrains.org/docs/automate-benchmark-experiments-analysis.md)
- [Add More Wandb Logging for Unsupervised Learning](https://docs.thousandbrains.org/docs/add-more-wandb-logging-for-unsupervised-learning.md): Improve logged information in unsupervised experiments (to wandb).
- [Add GPU Support for Monty](https://docs.thousandbrains.org/docs/add-gpu-support-for-monty.md): Utilize GPU acceleration for some of Monty's computations.
- [Use State Class Inside of LMs](https://docs.thousandbrains.org/docs/use-state-class-inside-of-lms.md): The CMP message format is used outside LMs to transmit messages but then converted back and forth to dictionaries within LMs.
- [Make Configs Easier to Use](https://docs.thousandbrains.org/docs/make-configs-easier-to-use.md): Use a standard configuration management library and simplify our configs.
- [Profile and Speed up Monty](https://docs.thousandbrains.org/docs/profile-and-optimize-monty.md): The current implementation of Montys algorithm is suboptimal and can be optimized. This item includes identifying computational bottlenecks and addressing them.
- [Find Faster Alternative to KDTree Search](https://docs.thousandbrains.org/docs/find-faster-alternative-to-kdtree-search.md): KD-Tree search is Monty's biggest speed bottleneck, so optimizing this would be beneficial.
- [OSS/Communication Improvements](https://docs.thousandbrains.org/docs/oss-communication-improvements.md): Things we would like to do for the open-source community or general communication.
- [Auto-Display Contributors in README](https://docs.thousandbrains.org/docs/auto-display-contributors-in-readme.md)
- [Better Documentation Platform](https://docs.thousandbrains.org/docs/better-documentation-platform.md)
- [Better Website Platform](https://docs.thousandbrains.org/docs/better-website-platform.md)
- [Build a Tutorial](https://docs.thousandbrains.org/docs/build-a-tutorial.md)
- [Find Forum Questions Themes & Improve Docs](https://docs.thousandbrains.org/docs/find-forum-questions-themes-improve-docs.md)
- [Insert Glossary Terms in Docs](https://docs.thousandbrains.org/docs/insert-glossary-terms-in-docs.md)
- [Make More Condensed Videos About the Project & Monty](https://docs.thousandbrains.org/docs/make-more-condensed-videos-about-the-project-monty.md)
- [Organize Local Meetup](https://docs.thousandbrains.org/docs/organize-local-meetup.md)
- [Organize & Start Podcast Series](https://docs.thousandbrains.org/docs/organize-start-podcast-series.md)
- [Robust Docs Preview Cleanup](https://docs.thousandbrains.org/docs/robust-docs-preview-cleanup.md)
- [Monty Tooling](https://docs.thousandbrains.org/docs/monty-tooling.md): These are efforts to develop tools that make Monty easier to use and build with.
- [Improve tbp.plot Visualizations During Learning and Inference](https://docs.thousandbrains.org/docs/improve-tbp-plot-visualizations-during-learning-and-inference.md): Improve the visualization tools available during learning and inference.
- [Monty Demos](https://docs.thousandbrains.org/docs/monty-demos.md): These are efforts to develop real-world demos of Monty's capabilities.
- [Improve Monty's Performance on Ultrasound](https://docs.thousandbrains.org/docs/improve-montys-performance-on-ultrasound.md): Improve the ability of Monty to learn and infer 3D objects in ultrasound images.
- [Theory Overview](https://docs.thousandbrains.org/docs/theory-overview.md): Intro page to the underlying theory of this project.
- [Compositional Models](https://docs.thousandbrains.org/docs/compositional-models.md): Our theory on how hierarchy can be used to learn objects composed of other objects.
- [Object Behaviors](https://docs.thousandbrains.org/docs/object-behaviors.md): Our theory on how object behaviors are represented, learned, and inferred.