Audits › Controls › Learns from its mistakes
AI and learning · control 19 of 21
Learns from its mistakes: does your trading bot have it?
Does it record the outcome of each decision and use it to correct itself (parameter tuning, trade memory, prompt or strategy changes)? If it self-corrects, does the change go through a review, a backtest or a limit before trading with it?
Why it matters
A bot that does not learn repeats the same mistake; one that rewrites itself without review can learn the wrong lesson with real money.
How to check it in the code
- Search for memory, reflection, feedback, outcome, retrain, optimize, hyperopt, self-improve.
- Check whether outcomes feed back into decisions automatically or only into reports.
- If changes are automatic, check for a gate: backtest, paper period, human approval, or bounded parameter ranges.
This is what the audit does for every bot in the list, on public code, without installing or running anything. Verdict values: present, partial, absent, not applicable, not verified. See the method.
Results across audited bots
Across 10 audited bots: 1 present, 3 partial, 2 absent, 2 not applicable, 2 not verified.
| Bot | Verdict | Evidence | Note |
|---|---|---|---|
| HKUDS/Vibe-Trading | Not verified | Directories agent/src/hypotheses/, agent/src/strategy_discovery/, agent/src/strategy_store/ exist | Names suggest a strategy hypothesis/self-correction subsystem but contents were not opened within budget; cannot confirm whether self-revisions pass through backtest/review before going live. |
| hummingbot/hummingbot | Not applicable | Same search as above, total_count 0 | No self-adjusting/AI learning loop found in this repo. |
| Drakkar-Software/OctoBot | Partial | deep_agent_team.py instructs agents to save insights to '/memories/trading_insights/', '/memories/signals/', and '/memories/distributions/' for future reference (long-term agent memory), and ai_index_trading.py logs full debate_history and judge_decisions when log_ai_decisions is enabled. | The bot records outcomes and lets the LLM write persistent memory that will influence future runs, but no code-level review, backtest, or limit gate on that self-written memory before it affects live decisions was found. |
| freqtrade/freqtrade | Not verified | ||
| jesse-ai/jesse | Present | README.md ML pipeline: explicit three-phase workflow — 'gather' (record_features/record_label during a backtest), 'train' (train_model() call, a separate manual mode), 'deploy' (ml_predict()/ml_predict_proba() inside the strategy) | Retraining requires a manual, separate step (running gather mode then train_model()); no evidence of the model rewriting itself online during a live session without review. |
| chrisleekr/binance-trading-bot | Absent | apps/web/src/features/backtest/components/backtest-llm-advisor.tsx ("On-demand config advisor for a finished run"); docs/concepts/notifiers.md, edge-decay-warning row | The advisor is triggered manually per finished backtest, not automatically. The one related signal, edge-decay-warning ("live results fall below the pinned backtest baseline"), is explicitly "Advisory only — the bot does NOT pause buys" and does not feed back into config automatically. No automatic retraining or unsupervised self-correction loop was found. |
| alsk1992/CloddsBot | Partial | docs/RISK_MANAGEMENT.md documents a DynamicKelly position-sizing module (src/trading/kelly.ts) that adjusts size from historical P&L, explicitly scoped as check #8 of 10 in the risk engine and marked 'No (adjusts size)' i.e. non-blocking — it cannot override the other deterministic limits. | Bounded, deterministic learning of position size exists; no evidence found of the agent rewriting its own strategy or prompts from outcomes, with or without a review gate. |
| Lumiwealth/lumibot | Partial | docs/AI_AGENT_MEMORY.md: agents can call `remember_lesson` ('stores proposed lessons by default. Pass outcome metadata with validated=True when a lesson has been validated by later outcome data') and theses receive daily 'thesis.outcome_observed' events. | |
| YizhiSong/FriesTrader | Absent | risk_rules.json thresholds are static and manually maintained (README First-time setup #2: 'review every other threshold -- the defaults here are illustrative, not a recommendation'). No code or spec text found (within the portion of PHASE_A_TASK.md/PHASE_B_TASK.md read) describing automatic parameter adjustment, prompt/strategy rewriting, or a backtest-gated self-correction loop based on trade_log.jsonl outcomes. | The bot's only 'memory' is the human reading trade_log.jsonl and manually editing risk_rules.json (README setup step 6). |
| c9s/bbgo | Not applicable | same search as decision_ia_filtrada | Not applicable: no self-adjusting/learning decision loop based on a model was found. |
If your bot does not have it
The watchdog enforces daily loss, drawdown, position size and heartbeat limits from outside the bot, with a read-only key, whatever the bot's own code does.