DEV Community

mattleeee
mattleeee

Posted on

Why "23 Altcoins Made +84%" Is Actually a Mirage

[{"type_of":"article","id":5077842,"title":"Overlap-Proof Cron: File Locks with O_EXCL and Stale-Lock Self-Healing","description":"Windows Task Scheduler has no concept of mutual exclusion between runs. If you tell it to fire a job...","published":true,"published_at":"2026-11-20T01:01:37.555Z","slug":"overlap-proof-cron-file-locks-with-oexcl-and-stale-lock-self-healing-2j6p","path":"/mattleeee/overlap-proof-cron-file-locks-with-oexcl-and-stale-lock-self-healing-2j6p","url":"https://dev.to/mattleeee/overlap-proof-cron-file-locks-with-oexcl-and-stale-lock-self-healing-2j6p","comments_count":0,"public_reactions_count":0,"page_views_count":10,"published_timestamp":"2026-11-20T01:01:37Z","body_markdown":"Windows Task Scheduler has no concept of mutual exclusion between runs. If you tell it to fire a job every 15 minutes, it will do exactly that, whether or not the previous instance is still alive. For a job that normally finishes in 10-30 seconds, this rarely matters. But when a stuck network call stretches a round to the timeout cap, two processes end up writing the same state.json concurrently, and each one silently overwrites the other's positions. The result is a corrupted state file and positions that vanish or double.\n\nThe fix is a lock file created with O_EXCL, plus a staleness rule so that a killed process can't wedge the scheduler forever. This post walks through the implementation, the failure modes, and a small test suite.\n\n## Why a lock file at all\n\nThe scheduler fires rounds on a fixed cadence. The job itself has no idea another instance is running. You have three broad options:\n\n1. Let the scheduler serialize. Windows Task Scheduler has a \"Do not start a new instance\" setting, but it only applies within the same task definition and is easy to misconfigure across task folders, service accounts, or manual runs. It also does nothing if someone triggers the script by hand.\n2. Use an in-process lock. Useless here: the two contenders are separate processes.\n3. Use an OS-level lock file. This works regardless of who started the process.\n\nThe lock file approach is the most portable and the easiest to reason about, provided you get two details right: atomic creation and stale-lock recovery.\n\n## Atomic creation with O_EXCL\n\nThe naive version of a lock is:\n\n

python\nif not os.path.exists(lock_path):\n open(lock_path, \"w\").close()\n run_job()\n

\n\nThis is a check-then-act race. Two processes can both observe that the file is absent, both create it, and both proceed. Under a 15-minute cadence the window is tiny, but \"tiny\" is not \"zero\", and the failure mode (concurrent writes to state.json) is exactly the one you're trying to prevent.\n\nO_EXCL closes the race at the kernel level. os.open(path, os.O_CREAT | os.O_EXCL | os.O_WRONLY) is a single atomic operation: it creates the file only if it does not already exist, and fails with FileExistsError otherwise. There is no window between the check and the create because there is no separate check.\n\n

python\nimport os\n\ndef try_acquire(lock_path: str, pid: int) -\u003e bool:\n try:\n fd = os.open(lock_path, os.O_CREAT | os.O_EXCL | os.O_WRONLY)\n except FileExistsError:\n return False\n try:\n os.write(fd, str(pid).encode(\"ascii\"))\n finally:\n os.close(fd)\n return True\n

\n\nWriting the PID is not strictly required for correctness, but it is invaluable when you're staring at a stuck lock at 2 a.m. and want to know whether the owning process is still alive.\n\n## The stale-lock problem\n\nO_EXCL gives you atomicity, but it also gives you a lock that never releases itself if the process dies. A taskkill, a crash, a power event, or a TerminateProcess from the scheduler all leave the lock file behind. The next scheduled round sees FileExistsError, exits, and the round after that does the same. The job is now permanently blocked, and nothing in the logs says why.\n\nYou need a rule for deciding when an existing lock is abandoned. The lock file's modification time is the natural arbiter. A run that normally takes 10-30 seconds and is capped by a timeout should never legitimately hold the lock for two hours. So:\n\n- If the lock is younger than the staleness threshold, treat it as live and skip this round.\n- If the lock is older than the threshold, treat it as abandoned, remove it, and try to acquire again.\n\nThe threshold has to be comfortably larger than the worst-case legitimate runtime. With a timeout cap of a few minutes, two hours is generous and still recovers within the same business day.\n\nThere is a subtle race in the takeover path: two processes can both decide the lock is stale, both unlink it, and both create a new one. The second unlink may fail with FileNotFoundError (fine), but the second O_EXCL create can still succeed after the first process has already created its own. To narrow this, do the takeover as a two-step: unlink, then attempt O_EXCL. If the O_EXCL fails, another process won the takeover and you should back off. It is not perfectly atomic, but the window is milliseconds and the consequence is a single overlapping round, not a persistent wedge. If you need stronger guarantees, rename the stale file to a unique name before unlinking and only proceed if the rename succeeded.\n\n## The full implementation\n\n

python\nimport os\nimport time\nimport logging\n\nlog = logging.getLogger(__name__)\n\nLOCK_PATH = r\"C:\\trading\\state.lock\"\nSTALE_SECONDS = 2 * 60 * 60 # 2 hours\n\n\ndef _lock_age_seconds(lock_path: str) -\u003e float:\n try:\n return time.time() - os.path.getmtime(lock_path)\n except FileNotFoundError:\n return float(\"inf\") # vanished; treat as stale so we retry\n\n\ndef acquire_lock(lock_path: str = LOCK_PATH,\n stale_seconds: int = STALE_SECONDS) -\u003e bool:\n \"\"\"Return True if this process owns the lock, False if another run holds it.\"\"\"\n pid = os.getpid()\n\n # Fast path: atomic exclusive create.\n try:\n fd = os.open(lock_path, os.O_CREAT | os.O_EXCL | os.O_WRONLY)\n except FileExistsError:\n pass\n else:\n try:\n os.write(fd, str(pid).encode(\"ascii\"))\n finally:\n os.close(fd)\n return True\n\n # Lock exists. Is it stale?\n age = _lock_age_seconds(lock_path)\n if age \u003c stale_seconds:\n log.warning(\"Lock held by another run (age %.1fs); skipping.\", age)\n return False\n\n log.warning(\"Stale lock detected (age %.1fs); attempting takeover.\", age)\n try:\n os.unlink(lock_path)\n except FileNotFoundError:\n pass # someone else already removed it\n\n # Second attempt; may still lose the race to another process.\n try:\n fd = os.open(lock_path, os.O_CREAT | os.O_EXCL | os.O_WRONLY)\n except FileExistsError:\n log.warning(\"Lost stale-lock takeover race; skipping.\")\n return False\n try:\n os.write(fd, str(pid).encode(\"ascii\"))\n finally:\n os.close(fd)\n return True\n\n\ndef release_lock(lock_path: str = LOCK_PATH) -\u003e None:\n try:\n os.unlink(lock_path)\n except FileNotFoundError:\n pass\n

\n\nUsage in the job entry point:\n\n

python\ndef main() -\u003e int:\n if not acquire_lock():\n return 0 # nothing to do; next scheduled round will pick it up\n try:\n run_trading_round()\n finally:\n release_lock()\n return 0\n

\n\nThe try/finally matters. If run_trading_round() raises, the lock must still be released, or the next round pays the staleness penalty for no reason.\n\n## Lock failure must not block the business path\n\nThere is a design decision hiding in the return 0 above. When the lock is held, the process exits cleanly without doing work. That is correct for a scheduled job: the next round is 15 minutes away, and the current holder is presumably doing the same work. But there is a second failure mode worth thinking about: what if the lock mechanism itself fails? A permissions change on the directory, an antivirus quarantine, a full disk that prevents file creation—these can all make os.open raise something other than FileExistsError.\n\nThe rule should be: if you cannot determine lock state due to an unexpected error, do not block the business path. Log loudly, then proceed. A duplicate run is bad; a job that silently stops running for days because a lock file couldn't be created is worse. In practice the trading logic has its own idempotency guards (position sizing against the current state file, for example), so a rare overlapping run is recoverable, while a permanently stalled scheduler is not.\n\n

python\ndef acquire_lock_or_proceed(lock_path: str = LOCK_PATH) -\u003e bool:\n \"\"\"Returns True if this run should proceed. Never raises.\"\"\"\n try:\n return acquire_lock(lock_path)\n except OSError:\n log.exception(\"Lock subsystem failed; proceeding without lock.\")\n return True\n

\n\nUse this wrapper at the top of main(). You lose strict serialization in the rare broken-lock case, which is the right trade.\n\n## Why not flock?\n\nPython's fcntl.flock is the idiomatic choice on POSIX, and it has a real advantage: the kernel releases the lock automatically when the owning process dies, so stale-lock recovery is free. On Windows, fcntl does not exist, and the portable substitutes are uneven:\n\n- msvcrt.locking works on file descriptors but has awkward semantics around byte ranges and blocking modes, and it does not release cleanly across all process-death scenarios.\n- portalocker wraps the platform differences reasonably well, but adds a dependency and its Windows backend still relies on msvcrt or LockFileEx via ctypes.\n- Third-party file-locking libraries vary in how they behave when the process is killed by the scheduler versus terminated by the user.\n\nO_EXCL sidesteps all of this. It is a single documented flag on every platform Python runs on, its semantics are identical everywhere, and the only thing you give up is automatic release on process death—which the mtime staleness rule handles. For a scheduled job that runs every 15 minutes, that trade is clearly worth it.\n\n## Testing the lock\n\nThree tests cover the behavior that matters. They run against a temporary directory and don't touch the real lock path.\n\n

python\nimport os\nimport time\nimport pytest\n\nfrom lock import acquire_lock, release_lock\n\n\[email protected]\ndef lock_path(tmp_path):\n return str(tmp_path / \"state.lock\")\n\n\ndef test_second_acquire_fails(lock_path):\n assert acquire_lock(lock_path) is True\n assert acquire_lock(lock_path) is False\n release_lock(lock_path)\n\n\ndef test_release_then_acquire_succeeds(lock_path):\n assert acquire_lock(lock_path) is True\n release_lock(lock_path)\n assert acquire_lock(lock_path) is True\n release_lock(lock_path)\n\n\ndef test_stale_lock_self_heals(lock_path):\n # Simulate a crashed process: lock file exists, mtime is old.\n with open(lock_path, \"w\") as f:\n f.write(\"99999\")\n old = time.time() - 3 * 60 * 60 # 3 hours\n os.utime(lock_path, (old, old))\n\n assert acquire_lock(lock_path, stale_seconds=2 * 60 * 60) is True\n release_lock(lock_path)\n

\n\nThe third test is the one that catches regressions in the takeover logic. It is easy to write a stale check that reads the file's mtime before checking existence, or that uses os.path.getmtime without handling the FileNotFoundError when another process unlinks in between. The test forces the stale path and verifies that acquisition succeeds.\n\nA fourth test worth having, though not strictly required: assert that a fresh lock is not taken over. Set the mtime to now and confirm acquire_lock returns False. That guards against a threshold bug that turns every lock into a stale lock.\n\n## Operational notes\n\n- Put the lock file on the same volume as the state file. Network paths reintroduce the race you just eliminated, because O_EXCL semantics on SMB shares are not always what you expect.\n- Log the PID and age whenever you skip a round. When the job silently stops doing work, that log line is the first thing you'll look for.\n- Keep the staleness threshold in configuration, not in code, so you can tighten it if the timeout cap changes.\n- The lock file should never be read by anything except the lock logic. Treat it as opaque.\n\nThe pattern generalizes: any scheduled job that mutates shared state and can occasionally overrun its interval needs exactly this. O_EXCL gives you the atomicity, the mtime rule gives you self-healing, and the fallback-to-proceed rule keeps a broken lock subsystem from becoming a broken scheduler.\n\nMore notes like this ship every week on this site.","positive_reactions_count":0,"cover_image":null,"tag_list":["filelock","oexcl","scheduledtasks","concurrency"],"canonical_url":"https://www.hkcode.dpdns.org/en/posts/107.html","reading_time_minutes":7,"user":{"name":"mattleeee","username":"mattleeee","twitter_username":null,"github_username":"mattleeee","user_id":4160029,"website_url":null,"profile_image":"https://media2.dev.to/dynamic/image/width=640,height=640,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png","profile_image_90":"https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png"}},{"type_of":"article","id":5052734,"title":"My Short Leg Was -37.9%, So I Cut the Strategy in Half","description":"Long-Short Decomposition: When the Short Leg Is the Reason Your Strategy Fails the...","published":true,"published_at":"2026-11-17T01:01:35.591Z","slug":"my-short-leg-was-379-so-i-cut-the-strategy-in-half-eh0","path":"/mattleeee/my-short-leg-was-379-so-i-cut-the-strategy-in-half-eh0","url":"https://dev.to/mattleeee/my-short-leg-was-379-so-i-cut-the-strategy-in-half-eh0","comments_count":0,"public_reactions_count":0,"page_views_count":0,"published_timestamp":"2026-11-17T01:01:35Z","body_markdown":"## Long-Short Decomposition: When the Short Leg Is the Reason Your Strategy Fails the Test\n\nThe backtest looked fine at a glance. EMA 20/50, dual-direction trend following, 7 majors, 4 years. Total return +265.8%. Then the t-statistic came back at 1.61 with p = 0.107. Positive, but not significant. A result you cannot defend in front of anyone who knows what a t-stat means.\n\nThe usual reaction is to go hunting for parameter tweaks: faster EMA, slower EMA, ATR stops, regime filters. All of that is premature. Before touching parameters, decompose the P\u0026L by direction. It costs twenty lines of code and it usually tells you more than a week of grid search.\n\nIn this case it told the whole story:\n\n| Leg | Return | Trades | t-stat |\n|---|---|---|---|\n| Long-only | +295.7% | 298 | — |\n| Short-only | -37.9% | 140 | -1.61 |\n| Combined | +265.8% | 438 | 1.61 |\n\nThe short leg did not just underperform. It carried the same magnitude of t-statistic as the combined system, with the opposite sign, and it dragged the aggregate from something you could argue about into the noise floor. The short book was not dead weight. It was actively destroying the evidence that the long book worked.\n\n## Why Decompose by Direction First\n\nA dual-direction system is two strategies stapled together with a shared parameter set. The entry logic is symmetric on paper — golden cross long, death cross short — but the return distributions on either side of zero are not symmetric at all. Aggregating them hides that asymmetry behind a single equity curve.\n\nDecomposition is the cheapest diagnostic you have. If one leg has a positive expectancy and the other has a negative one, the combined t-stat is a blend of two different processes and means very little. You are not testing \"does trend following work.\" You are testing \"does trend following work, averaged over two markets with different structural properties, weighted by trade count.\" That is a question nobody asked.\n\nThe fix is not to re-weight or re-optimize. It is to measure each leg on its own, then decide what the evidence actually supports.\n\n### Implementation: Same Engine, Filter by Sign\n\nThe key discipline is that you do not write a second backtester for the long-only variant. You run the exact same engine, record every trade with its direction, and slice the trade log afterward. This guarantees the two legs are directly comparable — same fills, same costs, same sizing.\n\n

python\nfrom dataclasses import dataclass\nimport numpy as np\nimport pandas as pd\n\n@dataclass\nclass Trade:\n symbol: str\n direction: int # +1 long, -1 short\n entry_time: pd.Timestamp\n exit_time: pd.Timestamp\n entry_px: float\n exit_px: float\n size: float # notional at entry\n\ndef trade_pnl(t: Trade, fee_bps: float = 5.0) -\u003e float:\n gross = t.direction * (t.exit_px - t.entry_px) / t.entry_px\n fees = 2 * fee_bps / 1e4\n return t.size * (gross - fees)\n\ndef run_engine(prices: dict[str, pd.DataFrame],\n fast: int = 20, slow: int = 50) -\u003e list[Trade]:\n \"\"\"Dual-direction EMA crossover. One position per symbol at a time.\"\"\"\n trades = []\n for sym, df in prices.items():\n ef = df[\"close\"].ewm(span=fast, adjust=False).mean()\n es = df[\"close\"].ewm(span=slow, adjust=False).mean()\n state = 0\n entry_px = entry_t = None\n for ts, f, s, px in zip(df.index, ef, es, df[\"close\"]):\n sig = 1 if f \u003e s else -1\n if state == 0:\n state, entry_px, entry_t = sig, px, ts\n elif sig != state:\n trades.append(Trade(sym, state, entry_t, ts,\n entry_px, px, size=1.0))\n state, entry_px, entry_t = sig, px, ts\n return trades\n

\n\nNow the decomposition itself. This is the part worth institutionalizing — put it in a shared module so every strategy report includes it by default.\n\n

python\ndef leg_stats(trades: list[Trade]) -\u003e pd.DataFrame:\n rows = []\n for leg, sign in [(\"long\", 1), (\"short\", -1)]:\n pnls = [trade_pnl(t) for t in trades if t.direction == sign]\n if len(pnls) \u003c 2:\n continue\n arr = np.array(pnls)\n tstat = arr.mean() / (arr.std(ddof=1) / np.sqrt(len(arr)))\n rows.append({\"leg\": leg, \"n\": len(arr),\n \"total\": arr.sum(), \"mean\": arr.mean(),\n \"t\": tstat})\n allp = np.array([trade_pnl(t) for t in trades])\n rows.append({\"leg\": \"combined\", \"n\": len(allp),\n \"total\": allp.sum(), \"mean\": allp.mean(),\n \"t\": allp.mean() / (allp.std(ddof=1) / np.sqrt(len(allp)))})\n return pd.DataFrame(rows).set_index(\"leg\")\n

\n\nTwo notes on the statistics. First, per-trade t-stats are not the same as a daily-returns t-stat on the equity curve; they answer slightly different questions. Per-trade is fine for a diagnostic like this, but if you want to publish a number, compute the t-stat on the daily P\u0026L series of each leg and let the trade-level view stay a screening tool. Second, if trades overlap across symbols, the independence assumption behind the t-stat is violated. For 7 majors running the same signal, correlation is high enough that the effective sample is smaller than 438. That makes the combined p = 0.107 optimistic, not pessimistic — another reason the original result never deserved the benefit of the doubt.\n\nFor completeness, the long-only variant is not a new engine. It is the same loop with the short branch suppressed:\n\n

python\ndef run_long_only(prices, fast=20, slow=50):\n trades = []\n for sym, df in prices.items():\n ef = df[\"close\"].ewm(span=fast, adjust=False).mean()\n es = df[\"close\"].ewm(span=slow, adjust=False).mean()\n in_pos, entry_px, entry_t = False, None, None\n for ts, f, s, px in zip(df.index, ef, es, df[\"close\"]):\n if not in_pos and f \u003e s: # golden cross enters\n in_pos, entry_px, entry_t = True, px, ts\n elif in_pos and f \u003c s: # death cross exits, no reverse\n trades.append(Trade(sym, 1, entry_t, ts, entry_px, px, 1.0))\n in_pos = False\n return trades\n

\n\nGolden cross enters, death cross exits, flat in between. No reverse, no short. The trade count drops from 438 to 298 — exactly the long leg of the dual-direction run, which is the check that your filter is doing what you think it is.\n\n## Why Shorting a Trend Follower Is Structurally Hard Here\n\nThe short leg was not unlucky. It was fighting three structural forces at once, and none of them show up in a parameter sweep.\n\n*Upward skew in the underlying.* Crypto majors over a four-year window spent most of their time in a positive drift regime. A trend follower's short entries cluster right after a death cross — that is, after a decline has already happened. In a market with positive drift, the expected value of being short immediately after a drawdown is negative even if the signal has some genuine momentum content. You are systematically selling after the move and buying back into the mean reversion.\n\n*Wick-heavy pops.* Crypto has no circuit breakers and thin books on the way up. Short positions get stopped or squeezed by single-bar wicks that leave no trace on a close-to-close backtest if your fills assume the close. The asymmetry is brutal: a long position caught in a wick-down is a drawdown, a short position caught in a wick-up is often a liquidation event at a price worse than any bar's close. Backtests systematically flatter shorts because they fill at benign prices.\n\n*Negative carry.* Funding on perpetuals is positive in bullish regimes, which means shorts pay to hold. Over four years that is a persistent bleed on top of the directional loss. If your backtest does not model funding, your short leg looks better than it is. If it does, the -37.9% is the honest number and it likely understates the real-world drag.\n\nPut together: the short side of a symmetric crossover is short a positively skewed, wick-prone, negative-carry asset class using a signal that enters late. That is not a parameter problem. No combination of 15/60 or 25/80 fixes the structure. This is why the correct response to a failing leg is usually deletion, not repair — the search space for \"repair\" is mostly overfitting in disguise.\n\n## The Discipline of Shipping Only What Passes\n\nDeleting the short leg raised the return to +295.7% on 298 trades. It did not raise the t-statistic into significance. Long-only is still positive-but-not-significant, and pretending otherwise would be the same mistake in a new costume.\n\nSo the label on the dashboard changed. The strategy is not promoted to live. It is downgraded to observation tier: a paper-trading observer that runs the long-only logic, logs every signal, and reports the same statistics on a rolling basis until the evidence accumulates. The dashboard text says exactly that — no green badge, no \"live\" tag, just the numbers and the tier.\n\nThe rule that makes this work is mechanical, not judgment-based:\n\n

python\ndef tier(t_stat: float, n: int, min_n: int = 200) -\u003e str:\n if n \u003c min_n:\n return \"observation\" # not enough evidence either way\n if abs(t_stat) \u003e= 2.0:\n return \"live-eligible\"\n if t_stat \u003e 0:\n return \"observation\" # positive but unproven\n return \"retired\"\n\ndef dashboard_row(leg: str, stats: pd.Series) -\u003e dict:\n return {\n \"leg\": leg,\n \"n\": int(stats[\"n\"]),\n \"total_pct\": round(100 * stats[\"total\"], 1),\n \"t\": round(float(stats[\"t\"]), 2),\n \"tier\": tier(float(stats[\"t\"]), int(stats[\"n\"])),\n \"note\": \"positive but not significant\" if stats[\"t\"] \u003c 2 and stats[\"t\"] \u003e 0 else \"\",\n }\n

\n\nThree properties matter here. The threshold is pre-registered, so you cannot move the goalposts after seeing the result. The tier is a pure function of the stats, so the dashboard cannot disagree with the data. And \"observation\" is a real state with real work attached — the observer keeps logging, and the strategy can only graduate by accumulating evidence, not by someone deciding it feels ready.\n\nParking the short leg as a paper observer rather than deleting it entirely is the second half of the discipline. You keep the measurement running at zero cost. If the regime shifts — funding flips persistently negative, or the market develops sustained downtrends — the short leg's stats will show it, and you will have four more years of honest data instead of a hunch. What you do not do is force it live because the long book needed a hedge, or because a symmetric strategy looks more elegant in a slide.\n\n## What Actually Changed\n\nThe final state: one engine, two legs measured separately, one leg deleted from live consideration, the survivor labeled honestly as unproven and parked in observation tier. The headline number went from +265.8% with a misleading aggregate t-stat to +295.7% with a clear statement that it is still not significant.\n\nThat is a worse-looking result than the original backtest and a much better one to build on. The decomposition did not find alpha. It found the reason the alpha was invisible, and then it refused to pretend the remainder was more than it was.\n\nMore notes like this ship every week on this site.\n\n---\n\n## Daily Picks\n\n\u003e The following pairs are selected from the multi-timeframe trend scanner (Gate.io futures) and are for technical-analysis study only — not investment advice.\n\u003e Data updated: 2026-11-17 08:46:11\n\n### Long\n\n| Pair | Signal | Price | Take Profit | Stop Loss | R/R |\n|------|--------|-------|-------------|-----------|-----|\n| LIT | 高质量标的做多 | $2.951 | $3.1458 | $2.8536 | 1:2.0 |\n\n_1 picks selected. Scanner runs every 15 minutes.","positive_reactions_count":0,"cover_image":null,"tag_list":["longshortdecomposition","symmetry","trendfollowing"],"canonical_url":"https://www.hkcode.dpdns.org/en/posts/104.html","reading_time_minutes":7,"user":{"name":"mattleeee","username":"mattleeee","twitter_username":null,"github_username":"mattleeee","user_id":4160029,"website_url":null,"profile_image":"https://media2.dev.to/dynamic/image/width=640,height=640,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png","profile_image_90":"https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png"}},{"type_of":"article","id":5044107,"title":"Three-Layer Strategy Validation and the Blind Spot That Bit Me","description":"Three-Layer Strategy Validation and the Blind Spot That Bit Me My validation stack had...","published":true,"published_at":"2026-11-16T01:01:32.227Z","slug":"three-layer-strategy-validation-and-the-blind-spot-that-bit-me-16f3","path":"/mattleeee/three-layer-strategy-validation-and-the-blind-spot-that-bit-me-16f3","url":"https://dev.to/mattleeee/three-layer-strategy-validation-and-the-blind-spot-that-bit-me-16f3","comments_count":0,"public_reactions_count":0,"page_views_count":10,"published_timestamp":"2026-11-16T01:01:32Z","body_markdown":"## Three-Layer Strategy Validation and the Blind Spot That Bit Me\n\nMy validation stack had three layers, and all three were green. DSR came out at 1.000. Coin-block Monte Carlo showed a 0% ruin rate across hundreds of randomized pool combinations. Split-half stability was positive in both halves of the trade sequence. I shipped it, and the strategy was garbage, because the backtest that produced those numbers had a lookahead bug that leaked future candle data into the entry decision.\n\nThe uncomfortable lesson: all three layers evaluate the statistical properties of one trade sequence. They assume that sequence was generated honestly. None of them audits how trades were generated. A poisoned sequence can be perfectly well-behaved statistically — in fact, a lookahead bug usually makes it more well-behaved, because the strategy is effectively peeking at the answer. High DSR, zero ruin, clean split-half — that's exactly what a leak looks like.\n\nThis post is the four-layer design I run now, with the code signatures and a table of what each layer actually catches. The fourth layer is the one that would have saved me.\n\n## Layer 1: Deflated Sharpe Ratio\n\nThe DSR (Bailey \u0026 López de Prado) adjusts an observed Sharpe ratio for the number of trials you ran, and for the non-normality of returns. If you tried 40 parameter sets and kept the best, the DSR asks: given 40 trials, skew, and kurtosis, is this Sharpe still distinguishable from the maximum you'd expect from noise?\n\n

python\nimport numpy as np\nfrom scipy.stats import norm\n\ndef deflated_sharpe_ratio(\n returns: np.ndarray,\n n_trials: int,\n sr_variance: float | None = None,\n periods_per_year: int = 252,\n) -\u003e float:\n \"\"\"\n Returns the probability that the true Sharpe exceeds zero,\n after deflating for n_trials and return non-normality.\n\n returns: per-period (e.g. daily) strategy returns, chronological\n n_trials: number of configurations you actually evaluated\n sr_variance: variance of Sharpe across your trials; if None,\n use the estimator below (conservative for small n_trials)\n \"\"\"\n r = np.asarray(returns, dtype=float)\n r = r[np.isfinite(r)]\n n = r.size\n if n \u003c 30 or n_trials \u003c 1:\n return 0.0\n\n mu, sd = r.mean(), r.std(ddof=1)\n if sd == 0:\n return 0.0\n sr = mu / sd # per-period Sharpe\n g1 = ((r - mu) ** 3).mean() / sd**3\n g2 = ((r - mu) ** 4).mean() / sd**4\n\n # Variance of the Sharpe estimator under skew/kurtosis\n sr_var = (1 - g1 * sr + (g2 - 1) / 4 * sr**2) / (n - 1)\n\n if sr_variance is None:\n # Expected max of n_trials draws from N(0, sr_var)\n e_max = np.sqrt(sr_var) * (\n (1 - np.euler_gamma) * norm.ppf(1 - 1 / n_trials)\n + np.euler_gamma * norm.ppf(1 - 1 / (n_trials * np.e))\n )\n else:\n e_max = np.sqrt(sr_variance) * norm.ppf(1 - 1 / n_trials)\n\n z = (sr - e_max) / np.sqrt(sr_var)\n return float(norm.cdf(z))\n

\n\nTwo things matter operationally. First, n_trials must be honest — every grid point, every filter variant, every \"quick check\" counts. Second, the DSR is only as good as the return series you feed it. It has no opinion on whether that series is real.\n\n## Layer 2: Coin-Block Monte Carlo\n\nI don't bootstrap individual trades. Trades cluster — volatility regimes, correlated entries, streaks. So I resample in blocks with random block boundaries, and I randomize the pool composition across hundreds of runs: which assets or sub-strategies are included, in what weights, drawn from the eligible universe. The output I care about is the ruin rate — the fraction of simulated paths that breach a drawdown or capital threshold.\n\n

python\ndef coin_block_monte_carlo(\n trade_pnls: np.ndarray,\n n_sims: int = 500,\n mean_block: int = 10,\n ruin_dd: float = 0.25,\n initial_capital: float = 1.0,\n rng: np.random.Generator | None = None,\n) -\u003e dict:\n \"\"\"\n Resample the trade sequence in random-length blocks and measure\n the fraction of paths that breach ruin_dd.\n\n trade_pnls: per-trade PnL as a fraction of capital, chronological\n mean_block: average block length (geometric block lengths)\n ruin_dd: drawdown threshold as fraction of peak equity\n \"\"\"\n rng = rng or np.random.default_rng(7)\n pnl = np.asarray(trade_pnls, dtype=float)\n n = pnl.size\n ruins, finals = 0, np.empty(n_sims)\n\n for s in range(n_sims):\n path, i = [], 0\n while len(path) \u003c n:\n blk = max(1, int(rng.geometric(1.0 / mean_block)))\n start = rng.integers(0, n)\n idx = (start + np.arange(blk)) % n # wrap-around blocks\n path.extend(pnl[idx])\n i += blk\n eq = initial_capital + np.cumsum(np.array(path[:n]))\n peak = np.maximum.accumulate(np.concatenate([[initial_capital], eq]))\n dd = 1.0 - eq / peak[1:]\n if dd.max() \u003e= ruin_dd:\n ruins += 1\n finals[s] = eq[-1]\n\n return {\n \"ruin_rate\": ruins / n_sims,\n \"median_final\": float(np.median(finals)),\n \"p05_final\": float(np.percentile(finals, 5)),\n }\n

\n\nThis layer answers: given this sequence's dependence structure, how fragile is the equity path? It does not answer whether the sequence could have been produced by the strategy in the first place. If the sequence is a fabrication, the Monte Carlo is resampling fiction with great rigor.\n\n## Layer 3: Split-Half Stability\n\nThe simplest and most brutal check: split the trade sequence in two, require positive expectation in both halves. I also require the halves to be roughly comparable in magnitude — a strategy whose entire edge lives in the first 30% of trades is a regime artifact, not a strategy.\n\n

python\ndef split_half_stability(\n trade_pnls: np.ndarray,\n min_trades_per_half: int = 30,\n max_decay_ratio: float = 0.5,\n) -\u003e dict:\n \"\"\"\n Require positive mean in both halves and that the second half\n retains at least max_decay_ratio of the first half's mean.\n\n trade_pnls: per-trade PnL, chronological\n \"\"\"\n pnl = np.asarray(trade_pnls, dtype=float)\n n = pnl.size\n if n \u003c 2 * min_trades_per_half:\n return {\"pass\": False, \"reason\": \"insufficient trades\"}\n\n h1, h2 = pnl[: n // 2], pnl[n // 2 :]\n m1, m2 = h1.mean(), h2.mean()\n ok = m1 \u003e 0 and m2 \u003e 0 and m2 \u003e= max_decay_ratio * m1\n return {\n \"pass\": bool(ok),\n \"mean_h1\": float(m1),\n \"mean_h2\": float(m2),\n \"decay_ratio\": float(m2 / m1) if m1 \u003e 0 else float(\"nan\"),\n }\n

\n\nSplit-half catches overfitting to a specific time window, edge decay, and single-regime dependence. It cannot catch a leak, because a leak inflates both halves. My lookahead bug made the second half look better than the first, and I read that as \"edge improving.\"\n\n## The Blind Spot\n\nHere's the thing all three layers share: they take trade_pnls as an axiom. Every one of them is a function of the sequence, not of the process that generated it. The DSR deflates for how many sequences you tried, not for whether each sequence was built from legal information. The Monte Carlo reshuffles the sequence. Split-half cuts it. None of them opens the strategy code and asks: at the moment of this entry decision, what data was visible?\n\nMy bug was mundane. The feature pipeline computed a rolling statistic with center=True on a window that extended past the decision timestamp, then joined features to signals on bar index. The backtest engine was happy to serve it. Every trade was entered with a few bars of future information baked into the feature. The resulting sequence had a Sharpe that survived 40 trials, zero ruin across 500 block simulations, and positive expectation in both halves. DSR = 1.000. Ruin = 0%. Both halves positive. All three green, all three meaningless.\n\nThe failure mode is structural: statistical validation is orthogonal to implementation correctness. You can have a flawless validation stack and a broken data path, and the stack will report success with total confidence.\n\n## Layer 4: Replay Testing\n\nThe fix is to stop trusting the backtest's trade sequence and instead force two independent code paths to agree on the same history. Replay testing drives the production trading code — the same signal generator, the same risk module, the same order logic that runs live — over real historical candles, bar by bar, with the data feed truncated at each decision timestamp. Then it diffs the resulting trades against the backtest's trades. Any divergence is a bug in one of the two, and the diff tells you where.\n\n

python\nfrom dataclasses import dataclass\nfrom typing import Callable, Iterable\n\n@dataclass\nclass Candle:\n ts: int\n open: float\n high: float\n low: float\n close: float\n volume: float\n\n@dataclass\nclass Trade:\n ts: int\n side: str\n qty: float\n price: float\n\ndef replay_production(\n candles: Iterable[Candle],\n on_candle: Callable[[list[Candle]], list[Trade]],\n warmup: int = 200,\n) -\u003e list[Trade]:\n \"\"\"\n Drive production code over history with a strictly causal feed.\n on_candle receives ONLY candles visible at the current timestamp.\n Returns the trade list the production code would have produced.\n \"\"\"\n history: list[Candle] = []\n trades: list[Trade] = []\n for c in candles:\n history.append(c)\n if len(history) \u003c= warmup:\n continue\n # The engine must never see beyond `history` here.\n trades.extend(on_candle(history))\n return trades\n\ndef compare_trade_streams(\n backtest: list[Trade],\n replay: list[Trade],\n ts_tolerance: int = 0,\n qty_rel_tol: float = 1e-6,\n) -\u003e dict:\n \"\"\"\n Diff two trade streams. Any mismatch is a validation failure,\n regardless of what layers 1-3 said.\n \"\"\"\n if len(backtest) != len(replay):\n return {\"match\": False, \"reason\": \"trade count differs\",\n \"backtest_n\": len(backtest), \"replay_n\": len(replay)}\n\n mismatches = []\n for i, (b, r) in enumerate(zip(backtest, replay)):\n if abs(b.ts - r.ts) \u003e ts_tolerance or b.side != r.side:\n mismatches.append({\"i\": i, \"kind\": \"timing_or_side\",\n \"backtest\": b, \"replay\": r})\n continue\n rel = abs(b.qty - r.qty) / max(abs(b.qty), 1e-12)\n if rel \u003e qty_rel_tol:\n mismatches.append({\"i\": i, \"kind\": \"qty\",\n \"backtest\": b, \"replay\": r})\n\n return {\"match\": not mismatches, \"mismatches\": mismatches[:20],\n \"n_mismatches\": len(mismatches)}\n

\n\nThe critical property is causality enforcement. The replay harness hands the production code a list that ends at the current bar — there is no future data in the object, so a center=True bug becomes a crash or an obviously wrong feature, not a silent edge. If the production code needs a warmup window, it gets only the past. If it tries to index forward, it raises.\n\nTwo practical notes from running this:\n\n1. Diff at the trade level, not the signal level. Signals can diverge harmlessly due to float ordering. Trades — timestamp, side, quantity — must match. A single-trade mismatch is worth a full investigation.\n2. Run replay on the same candles the backtest used, byte for byte. If the backtest reads from a research store and replay reads from a production feed, you're testing data plumbing, not strategy logic. Pin one dataset.\n\nWhen I added this layer, the lookahead bug surfaced in minutes. The replay trade stream diverged from the backtest at the first signal after warmup, and the diff pointed straight at the feature join. The first three layers had been reporting success on a sequence the production code could never have generated.\n\n## What Each Layer Catches\n\n| Layer | Catches | Does not catch |\n|---|---|---|\n| DSR | Multiple-testing inflation, non-normal Sharpe, selection bias across trials | Lookahead, data leakage, execution mismatches, regime fragility |\n| Coin-block Monte Carlo | Path fragility, drawdown/ruin risk under dependence, pool-composition sensitivity | Whether the trade sequence is achievable by production code |\n| Split-half | Edge decay, single-regime dependence, time-window overfitting | Leaks (they inflate both halves), implementation divergence |\n| Replay | Lookahead, engine/production divergence, data-path bugs, timestamp handling | Overfitting, multiple testing, ruin risk, regime dependence |\n\nThe layers are complementary, and the order matters less than the coverage. Layers 1–3 audit the statistics of the sequence. Layer 4 audits the provenance of the sequence. You need both, and the provenance check is the one that's cheap to skip and expensive to omit.\n\nThe general principle: every validation layer should be tested against a deliberately broken strategy. Feed your DSR a known-leaky sequence — it should pass, and now you know its blind spot. Feed your replay harness a strategy with a one-bar peek — it should fail loudly. A validation layer you've never seen fail is a layer you don't understand.\n\nMore notes like this ship every week on this site.\n\n---\n\n## Daily Picks\n\n\u003e The following pairs are selected from the multi-timeframe trend scanner (Gate.io futures) and are for technical-analysis study only — not investment advice.\n\u003e Data updated: 2026-11-16 08:46:12\n\n### Long\n\n| Pair | Signal | Price | Take Profit | Stop Loss | R/R |\n|------|--------|-------|-------------|-----------|-----|\n| SOXL | 高质量标的做多 | $57.98 | $59.8354 | $56.8204 | 1:1.6 |\n\n### Short\n\n| Pair | Signal | Price | Take Profit | Stop Loss | R/R |\n|------|--------|-------|-------------|-----------|-----|\n| EVAA | 热点跟涨 | $2.7639 | $2.6257 | $2.8606 | 1:1.4 |\n\n_2 picks selected. Scanner runs every 15 minutes.
","positive_reactions_count":0,"cover_image":null,"tag_list":["dsr","montecarlo","splithalf","strategyvalidation"],"canonical_url":"https://www.hkcode.dpdns.org/en/posts/103.html","reading_time_minutes":8,"user":{"name":"mattleeee","username":"mattleeee","twitter_username":null,"github_username":"mattleeee","user_id":4160029,"website_url":null,"profile_image":"https://media2.dev.to/dynamic/image/width=640,height=640,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png","profile_image_90":"https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png"}},{"type_of":"article","id":4987038,"title":"Anti-Overfitting for Retail Backtests: Few Rules, Fewer Trades, More Skepticism","description":"Why the Fifth Rule Is Usually Noise Every discretionary backtest dies the same death. You...","published":true,"published_at":"2026-11-06T08:30:51.347Z","slug":"anti-overfitting-for-retail-backtests-few-rules-fewer-trades-more-skepticism-5e4o","path":"/mattleeee/anti-overfitting-for-retail-backtests-few-rules-fewer-trades-more-skepticism-5e4o","url":"https://dev.to/mattleeee/anti-overfitting-for-retail-backtests-few-rules-fewer-trades-more-skepticism-5e4o","comments_count":0,"public_reactions_count":0,"page_views_count":10,"published_timestamp":"2026-11-06T08:30:51Z","body_markdown":"## Why the Fifth Rule Is Usually Noise\n\nEvery discretionary backtest dies the same death. You start with a simple idea — buy the pullback, exit on a failed retest. It returns something mediocre. So you add a filter: skip trades when volatility is too high. The equity curve improves. You add another: only take longs when the trend filter agrees. Better. One more: require volume confirmation. Now the curve looks great, and you have five conditions, each of which was chosen because it made the past look better.\n\nThat is the entire failure mode in one sentence. Each added rule is a new degree of freedom, and every degree of freedom is a chance to fit noise rather than signal. A retail backtest with ten boolean conditions and a parameter grid is not a strategy; it is a compression algorithm for historical data. The compression is excellent. The generalization is not.\n\nThe defensive doctrine here is short enough to memorize:\n\n- Few rules over many. Every added condition must survive its own audit.\n- Fewer trades with clear invalidation over many marginal ones. A trade you cannot invalidate is not a trade, it is a hope.\n- Distrust any edge that only appears after tuning. If it needs the search to exist, the search created it.\n- Re-test on unseen data before believing anything. Including your own improvements.\n\nBelow is how to apply that in code and process.\n\n## The Rule That \"Improved\" the Backtest by Rejecting 52 Trades\n\nA concrete case. The base strategy took ~400 trades over several years and had a modest expectancy. Adding a filter — \"skip entries when the 20-day realized volatility is above its 80th percentile\" — removed 52 trades and lifted the profit factor from 1.18 to 1.41. Tempting. Ship it.\n\nBut that filter was not proposed in advance. It was proposed after looking at the losing trades, noticing they clustered in high-volatility regimes, and encoding that observation as a rule. That is a hypothesis generated from the same data used to test it. The 52 rejected trades are not out-of-sample evidence; they are the training set for the filter.\n\nThe fix is an A/B audit of the rule itself, treated as a separate hypothesis with its own test:\n\n

python\nimport numpy as np\nimport pandas as pd\n\ndef ab_audit(trades: pd.DataFrame, filter_mask: pd.Series,\n n_perm: int = 5000, seed: int = 7) -\u003e dict:\n \"\"\"\n trades: one row per trade with columns ['pnl', 'entry_time'].\n filter_mask: True where the filter would REJECT the trade.\n Question: is the rejected subset's mean pnl meaningfully worse than\n the kept subset, beyond what random rejection of the same size gives?\n \"\"\"\n rng = np.random.default_rng(seed)\n pnl = trades[\"pnl\"].to_numpy()\n rejected = filter_mask.to_numpy()\n k = rejected.sum()\n\n observed_gap = pnl[~rejected].mean() - pnl[rejected].mean()\n\n gaps = np.empty(n_perm)\n for i in range(n_perm):\n idx = rng.choice(len(pnl), size=k, replace=False)\n mask = np.zeros(len(pnl), dtype=bool)\n mask[idx] = True\n gaps[i] = pnl[~mask].mean() - pnl[mask].mean()\n\n p_value = (gaps \u003e= observed_gap).mean()\n return {\n \"n_trades\": len(pnl),\n \"n_rejected\": int(k),\n \"observed_gap\": round(observed_gap, 4),\n \"p_value\": round(p_value, 4),\n }\n

\n\nIf the p-value is not small — and with 52 trades out of 400, it usually is not — the filter is not evidence of a regime effect. It is a random subset that happened to contain losers. The honest outcome is to drop the rule and keep the simpler system.\n\nThere is a second, cheaper audit: does the rule survive on a holdout period it never saw? Split by time, not randomly. Random splits leak regime information across the boundary; time splits mimic what you will actually face.\n\n

python\ndef time_split_ab(trades: pd.DataFrame, filter_mask: pd.Series,\n split_frac: float = 0.6) -\u003e pd.DataFrame:\n trades = trades.sort_values(\"entry_time\").reset_index(drop=True)\n filter_mask = filter_mask.loc[trades.index]\n cut = int(len(trades) * split_frac)\n\n rows = []\n for name, sl in [(\"in_sample\", slice(0, cut)), (\"holdout\", slice(cut, None))]:\n sub = trades.iloc[sl]\n rej = filter_mask.iloc[sl]\n rows.append({\n \"period\": name,\n \"pf_all\": profit_factor(sub[\"pnl\"]),\n \"pf_kept\": profit_factor(sub.loc[~rej, \"pnl\"]),\n \"n_rejected\": int(rej.sum()),\n })\n return pd.DataFrame(rows)\n\n\ndef profit_factor(pnl: pd.Series) -\u003e float:\n wins = pnl[pnl \u003e 0].sum()\n losses = -pnl[pnl \u003c 0].sum()\n return round(wins / losses, 3) if losses \u003e 0 else float(\"inf\")\n

\n\nA rule that lifts profit factor in-sample but does nothing in the holdout is a rule you invented, not one the market gave you.\n\n## The Multiple-Comparisons Trap in Parameter Sweeps\n\nParameter sweeps are where retail backtests quietly become data mining. Run 12 variants of a stop-loss, 8 of a take-profit, 5 of a trend filter — that is 480 configurations. The best one will look good even if every configuration is pure noise. The maximum of 480 random draws is not a discovery; it is the expected maximum of 480 random draws.\n\nYou can quantify this. Under the null hypothesis of no edge, the best in-sample Sharpe across N independent trials has an expected value roughly on the order of sqrt(2 * ln(N)) times the standard error of a single trial. That is why \"I tested 500 combos and found one with Sharpe 2.1\" is not impressive.\n\nTwo practical defenses:\n\n*1. Correct the significance threshold.* If you test N configurations and want a family-wise error rate of α, require each individual result to beat a Bonferroni-style threshold of α/N. Crude, but it kills most false discoveries outright.\n\n

python\ndef sweep_report(results: pd.DataFrame, alpha: float = 0.05) -\u003e pd.DataFrame:\n \"\"\"\n results: columns ['config_id', 'sharpe', 'n_trades', 'p_value']\n p_value: from a permutation test of that config's expectancy vs zero.\n \"\"\"\n n = len(results)\n threshold = alpha / n\n out = results.copy()\n out[\"bonferroni_threshold\"] = threshold\n out[\"survives\"] = out[\"p_value\"] \u003c threshold\n return out.sort_values(\"p_value\")\n

\n\n*2. Count your trials honestly.* Every variant you ran and discarded is a trial. If you cannot remember how many you ran, your effective N is larger than you think, and your best result is weaker than it looks. Keep a log.\n\n

python\nimport json, datetime\n\ndef log_trial(path: str, config: dict, metrics: dict) -\u003e None:\n with open(path, \"a\") as f:\n f.write(json.dumps({\n \"ts\": datetime.datetime.utcnow().isoformat(),\n \"config\": config,\n \"metrics\": metrics,\n }) + \"\\n\")\n

\n\nThe log is uncomfortable to keep precisely because it makes the search visible. That is the point.\n\n## Honest Metrics\n\nMost retail backtests report the wrong numbers. Total return and win rate are the worst offenders — win rate in particular rewards strategies that take many small wins and one catastrophic loss. Report these instead:\n\n### Expectancy net of fees\n\nPer-trade expectancy after realistic costs, including slippage. If your edge disappears at 2x your assumed fees, it was never an edge.\n\n

python\ndef expectancy(pnl: pd.Series, fees_per_trade: float) -\u003e float:\n net = pnl - fees_per_trade\n return round(net.mean(), 4)\n

\n\n### Profit factor\n\nGross wins over gross losses. Above ~1.3 net of costs is worth a second look; below 1.1 is noise territory. But profit factor alone hides path dependence — pair it with drawdown.\n\n### Monthly consistency\n\nBucket returns by calendar month. A strategy that makes all its money in three months out of forty-eight is a regime bet, not a system.\n\n

python\ndef monthly_consistency(returns: pd.Series) -\u003e pd.DataFrame:\n monthly = (1 + returns).resample(\"ME\").prod() - 1\n return pd.DataFrame({\n \"month\": monthly.index.strftime(\"%Y-%m\"),\n \"return\": monthly.round(4),\n \"positive\": monthly \u003e 0,\n })\n

\n\nLook at the fraction of positive months and the worst month. If 80% of the total return comes from the top 10% of months, you are not running a strategy; you are waiting for a handful of lucky regimes and paying fees the rest of the time.\n\n## The Person Who Built It Cannot Approve It\n\nThis is the cultural rule, and it matters more than any code above. The person who tuned the parameters has a vested interest in the curve. They have seen every variant. They know which filter \"works.\" That knowledge is contamination, and it does not wash off.\n\nThe fix is procedural, not technical: the backtester must never be the approver. Someone else — a partner, a friend, a second account with a written checklist — reviews the out-of-sample report without seeing the parameter search. They get the final config, the holdout results, and the trial count. Nothing else. If the strategy only convinces the person who built it, it does not convince anyone.\n\nA minimal approval checklist:\n\n- Rule count is small and each rule has an A/B audit with a p-value.\n- The trial log exists and the best result survives a multiple-comparisons correction.\n- Expectancy is positive after 2x assumed fees.\n- Profit factor and monthly consistency hold on the holdout period.\n- A second person, who did not tune it, signed off.\n\nIf any line fails, the strategy goes back to the simplest version that passed. Usually that is the version with two rules and a lot of skipped trades — which is exactly the version most people discard on the way to a prettier curve.\n\nMore notes like this ship every week on this site.\n\n---\n\n## Daily Picks\n\n\u003e The following pairs are selected from the multi-timeframe trend scanner (Gate.io futures) and are for technical-analysis study only — not investment advice.\n\u003e Data updated: 2026-11-06 08:46:43\n\n### Short\n\n| Pair | Signal | Price | Take Profit | Stop Loss | R/R |\n|------|--------|-------|-------------|-----------|-----|\n| MUU | 庄家做空 | $13.57 | $12.593 | $14.045 | 1:2.1 |\n| KOMA | 短平快做空 | $0.0128 | $0.0124 | $0.0131 | 1:1.2 |\n\n_2 picks selected. Scanner runs every 15 minutes.","positive_reactions_count":0,"cover_image":null,"tag_list":["overfitting","backtestdiscipline","outofsample","rulecount"],"canonical_url":"https://www.hkcode.dpdns.org/en/posts/39.html","reading_time_minutes":6,"user":{"name":"mattleeee","username":"mattleeee","twitter_username":null,"github_username":"mattleeee","user_id":4160029,"website_url":null,"profile_image":"https://media2.dev.to/dynamic/image/width=640,height=640,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png","profile_image_90":"https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png"}},{"type_of":"article","id":4987034,"title":"Whale-Pump Altcoins and the Pullback Problem","description":"The Anatomy of an Orchestrated Pump A whale-driven altcoin pump is not a random walk with...","published":true,"published_at":"2026-11-06T08:30:15.143Z","slug":"whale-pump-altcoins-and-the-pullback-problem-loo","path":"/mattleeee/whale-pump-altcoins-and-the-pullback-problem-loo","url":"https://dev.to/mattleeee/whale-pump-altcoins-and-the-pullback-problem-loo","comments_count":0,"public_reactions_count":0,"page_views_count":30,"published_timestamp":"2026-11-06T08:30:15Z","body_markdown":"## The Anatomy of an Orchestrated Pump\n\nA whale-driven altcoin pump is not a random walk with fat tails. It has a schedule. Understanding that schedule is the difference between a pullback filter that protects you and one that quietly starves your strategy.\n\nThe four phases look like this:\n\n*Accumulation.* Price ranges tight, often for days. On low timeframes you see repeated sweeps of the same high with no follow-through. Volume is thin and inconsistent. The operator is absorbing supply and building a position without moving price. Retail sees a boring chart.\n\n*Markup.* This is where the structure breaks. Price leaves the range in continuous legs — impulse, shallow dip, impulse, shallow dip. The dips are engineered to be just deep enough to shake out weak hands and just short enough that anyone waiting for a \"proper\" retracement never gets filled. Volume expands on the legs and contracts on the dips. This is the phase most pullback strategies are built for, and the phase where they fail.\n\n*Distribution.* The operator needs exit liquidity. The buy-side liquidity created by the markup — stop orders above highs, breakout traders, FOMO limit orders — is the target. Price often pushes one final leg into that liquidity, then stalls. Ranges form near the highs. The dips get deeper and the recoveries get weaker. By the time a textbook pullback signal prints, you are buying into the distribution, not the trend.\n\n*Collapse.* Once the operator is out, the bid disappears. The same continuous legs that went up go down, faster, because there is no absorption underneath.\n\nThe critical point: the markup and the distribution look similar on a single candle and completely different across a sequence. Your pullback filter has to distinguish them, and most don't.\n\n## Why Pullback Quality Degrades in Pump Regimes\n\nA classic pullback entry says: wait for three consecutive red candles, a retracement to a moving average or a Fibonacci level, then buy the first green confirmation. This works in liquid, mean-reverting markets where participants are numerous and independent. In an orchestrated pump, it breaks for structural reasons:\n\n1. The dips are too shallow. A continuous-leg markup might retrace 15–25% of the prior leg on the 15m chart. A three-candle pullback filter calibrated on BTC or ETH expects 40–60%. You get no entries during the entire profitable body of the move.\n\n2. The dips are too fast. When the operator does allow a deeper dip, it happens in one or two violent candles, not a clean three-candle sequence. Your filter sees a single red candle followed by a green one and rejects it as \"not enough confirmation.\"\n\n3. The liquidity is manufactured. The pullback itself creates the buy-side liquidity the operator needs to distribute into. A pullback that looks clean on the chart is, in this regime, often the signal that distribution is starting. The filter is not just late — it is systematically inverted at the top.\n\n4. Volatility is non-stationary. The same coin can have a 2% average true range during accumulation and a 12% ATR during markup. A fixed pullback depth threshold is meaningless across those regimes.\n\nThe result in backtests is a filter that rejects the middle of the move (too shallow) and accepts the top (deep enough, but for the wrong reason). It looks conservative. It is actually selective in the worst possible way.\n\n## What the Backtest Data Showed\n\nI ran a simple momentum strategy on a basket of mid-cap altcoins over eighteen months, tagging each entry by phase using a manual label set (accumulation / markup / distribution / collapse) derived from volume and range behavior. The strategy had two variants:\n\n- Strict: require a three-candle pullback with a retracement of at least 38% of the prior leg, plus a volume contraction on the pullback.\n- Relaxed: require a single-candle dip below the prior leg's midpoint, no volume condition, with a hard invalidation below the leg low.\n\nThe strict variant produced roughly 40% of the trade count of the relaxed variant. More importantly, the distribution of returns was inverted relative to expectation:\n\n| Variant | Trades | Win rate | Avg R (markup) | Avg R (distribution) |\n|---|---|---|---|---|\n| Strict | 112 | 51% | +0.9 | −1.4 |\n| Relaxed | 287 | 44% | +1.3 | −1.1 |\n\nThe strict filter missed most of the markup entries because the dips were too shallow, and it still got caught in distribution because the deeper dips near the top satisfied its retracement threshold. The relaxed version took more trades, won less often, but captured a larger share of the markup body and — critically — its invalidation cut losses faster in distribution.\n\nThe conclusion was not \"loosen everything.\" It was: the confirmation step was doing the wrong job. It was supposed to filter out bad entries, but in this regime the confirmation was filtering out good ones and letting bad ones through. The invalidation step was doing almost none of the work.\n\n## The Practical Response\n\nThree changes, applied together. None of them is optional if you want the others to work.\n\n### 1. Relax the confirmation\n\nStop requiring a multi-candle pullback. In a continuous-leg markup, the entry signal is the resumption, not the depth of the dip. A single candle that dips below the prior leg's midpoint and closes back above it is enough. You are trading the operator's need to keep the leg going, not a textbook retracement.\n\n

python\ndef pullback_entry(df, leg_lookback=8, dip_frac=0.5):\n \"\"\"\n df: OHLCV DataFrame with columns open, high, low, close, volume.\n Returns a boolean Series: True on bars where a relaxed pullback entry triggers.\n \"\"\"\n # Identify the most recent impulse leg: highest high over lookback vs lowest low before it\n leg_high = df[\"high\"].rolling(leg_lookback).max()\n leg_low = df[\"low\"].rolling(leg_lookback).min()\n leg_mid = leg_low + (leg_high - leg_low) * dip_frac\n\n # Dip below midpoint and close back above it on the same bar\n dipped = df[\"low\"] \u003c leg_mid\n reclaimed = df[\"close\"] \u003e leg_mid\n green = df[\"close\"] \u003e df[\"open\"]\n\n # Require the leg to be an actual impulse: range expansion vs prior window\n prior_range = (df[\"high\"] - df[\"low\"]).rolling(leg_lookback * 2).mean().shift(leg_lookback)\n impulse = (leg_high - leg_low) \u003e 1.5 * prior_range\n\n return dipped \u0026 reclaimed \u0026 green \u0026 impulse\n

\n\nThe impulse condition matters. Without it, you are buying every dip in every range. With it, you are only buying dips inside a leg that has already shown range expansion.\n\n### 2. Tighten the invalidation\n\nBecause the confirmation is weaker, the invalidation must be harder. Place it below the leg low, not below the pullback low, and make it a hard stop with no discretion. In a pump regime, a break of the leg low is the operator telling you the markup is over. Respect it immediately.\n\n

python\ndef apply_invalidation(df, entries, leg_lookback=8):\n \"\"\"\n For each entry, set stop at the low of the impulse leg.\n Returns a Series of stop prices aligned to entry bars (NaN elsewhere).\n \"\"\"\n leg_low = df[\"low\"].rolling(leg_lookback).min()\n stops = leg_low.where(entries)\n return stops\n\ndef position_size(equity, entry_price, stop_price, risk_frac=0.005):\n \"\"\"\n Size down: risk_frac is deliberately small because regime certainty is low.\n \"\"\"\n risk_per_unit = entry_price - stop_price\n if risk_per_unit \u003c= 0:\n return 0.0\n return (equity * risk_frac) / risk_per_unit\n

\n\nNote the risk_frac=0.005. That is half a percent of equity per trade, not the 1–2% you might use in a liquid market. The reason is in the next section.\n\n### 3. Size down because regime certainty is lower\n\nThe relaxed confirmation raises your trade count and lowers your per-trade edge. The tighter invalidation caps your loss per trade but also increases the frequency of stops — you will get shaken out of legs that continue. Both effects mean your estimate of the win rate is noisier and your drawdowns are more frequent.\n\nThe correct response is not to compensate with leverage. It is to reduce risk per trade and let the larger sample of trades do the work. A strategy with a 44% win rate and a 1.3R average winner is profitable, but only if you survive the losing streaks. Half a percent per trade gives you room for a 15-trade losing streak without a 10% drawdown.\n\n

python\ndef regime_adjusted_risk(phase, base_risk=0.005):\n \"\"\"\n phase: 'accumulation' | 'markup' | 'distribution' | 'collapse'\n Scale risk by how much you trust the regime label.\n \"\"\"\n scale = {\n \"accumulation\": 0.5, # low volatility, but no trend yet\n \"markup\": 1.0, # the regime the strategy is built for\n \"distribution\": 0.3, # structure is degrading, cut exposure\n \"collapse\": 0.0, # do not trade\n }\n return base_risk * scale.get(phase, 0.0)\n

\n\nThe phase label can be approximate. You don't need a perfect classifier; you need a coarse filter that stops you from sizing up in distribution. Volume contraction on legs, weakening recovery highs, and range compression near the top are enough to flag it.\n\n## What Changes in Practice\n\nThe strict pullback filter felt safe. It was not. It rejected the body of the move and accepted the top, because it was measuring the wrong thing — depth of retracement instead of continuation of structure.\n\nThe fix is a trade-off, not a free lunch. You take more trades, win less often, and rely on a hard stop and smaller size to keep the tail controlled. In a pump-driven market, that is the honest shape of the edge: you are not predicting the operator's next move, you are participating in the leg while it lasts and getting out when the structure breaks.\n\nMore notes like this ship every week on this site.","positive_reactions_count":0,"cover_image":null,"tag_list":["marketstructure","altcoins","whales","momentum"],"canonical_url":"https://www.hkcode.dpdns.org/en/posts/38.html","reading_time_minutes":6,"user":{"name":"mattleeee","username":"mattleeee","twitter_username":null,"github_username":"mattleeee","user_id":4160029,"website_url":null,"profile_image":"https://media2.dev.to/dynamic/image/width=640,height=640,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png","profile_image_90":"https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png"}},{"type_of":"article","id":4987028,"title":"Market Sentiment as an Input: Integrating the Fear \u0026 Greed Index","description":"The Fear \u0026amp; Greed index is a daily composite scaled 0–100, built from volatility, momentum, social...","published":true,"published_at":"2026-11-06T08:29:38.953Z","slug":"market-sentiment-as-an-input-integrating-the-fear-greed-index-1m4o","path":"/mattleeee/market-sentiment-as-an-input-integrating-the-fear-greed-index-1m4o","url":"https://dev.to/mattleeee/market-sentiment-as-an-input-integrating-the-fear-greed-index-1m4o","comments_count":0,"public_reactions_count":0,"page_views_count":10,"published_timestamp":"2026-11-06T08:29:38Z","body_markdown":"The Fear \u0026 Greed index is a daily composite scaled 0–100, built from volatility, momentum, social signals, dominance and trend components. The temptation is to trade it directly: buy at 10, sell at 90. That rarely survives contact with real data — the index spends weeks in \"extreme\" territory during trending markets, and each reading is a lagging summary of moves that already happened.\n\nThe more defensible use is as a context layer. You fetch it once a day, combine it with market internals you already track, and let it bias — not dictate — your execution. In practice that means three things:\n\n1. A scheduled job that pulls the index plus internals and computes a composite score.\n2. A morning summary pushed to wherever you read it.\n3. A soft filter that suppresses new entries in the direction the crowd is already leaning, while leaving exits and existing positions alone.\n\nThis post walks through the pipeline, the scoring design, the logging that lets you test whether the filter deserves to exist, and the discipline of killing it when the data says so.\n\n## The data source and caching\n\nThe public Fear \u0026 Greed endpoints (crypto and equities variants) return a small JSON payload: a value, a classification string, and a timestamp. Two rules apply to any external sentiment source:\n\n- Cache aggressively. The index updates once a day. Hitting the endpoint on every strategy tick is wasteful and gets you rate-limited.\n- Never let a fetch failure break the trading loop. A missing sentiment value should degrade to neutral, not crash the process.\n\nA simple file-backed cache with a TTL and a stale-fallback path covers both:\n\n

python\nimport json\nimport time\nfrom datetime import datetime, timezone\nfrom pathlib import Path\n\nimport requests\n\nCACHE_PATH = Path(\"data/fng_cache.json\")\nTTL_SECONDS = 6 * 60 * 60\nENDPOINT = \"https://api.alternative.me/fng/?limit=1\"\n\ndef fetch_fng(timeout: float = 10.0) -\u003e dict | None:\n \"\"\"Return {'value': int, 'label': str, 'ts': iso} or None on failure.\"\"\"\n if CACHE_PATH.exists():\n cached = json.loads(CACHE_PATH.read_text())\n age = time.time() - cached[\"fetched_at\"]\n if age \u003c TTL_SECONDS:\n return cached[\"payload\"]\n\n try:\n resp = requests.get(ENDPOINT, timeout=timeout)\n resp.raise_for_status()\n row = resp.json()[\"data\"][0]\n payload = {\n \"value\": int(row[\"value\"]),\n \"label\": row[\"value_classification\"],\n \"ts\": datetime.fromtimestamp(\n int(row[\"timestamp\"]), tz=timezone.utc\n ).isoformat(),\n }\n CACHE_PATH.parent.mkdir(parents=True, exist_ok=True)\n CACHE_PATH.write_text(json.dumps(\n {\"fetched_at\": time.time(), \"payload\": payload}\n ))\n return payload\n except Exception:\n # Fall back to whatever is on disk, even if stale.\n if CACHE_PATH.exists():\n return json.loads(CACHE_PATH.read_text())[\"payload\"]\n return None\n

\n\nThe stale fallback matters more than it looks. On a network blip you'd rather filter on yesterday's reading than skip the filter entirely and take a full-size entry into extreme greed.\n\nNote that the timestamp in the payload is the index's own publication time, not your fetch time. Store both. You'll want the index timestamp when you later align sentiment with trade timestamps.\n\n## Designing the composite score\n\nThe raw index is one number. A composite that blends it with internals you trust is more stable and more explainable. The design goal is not to find the single best score — it's to build something whose components you can inspect when it misbehaves.\n\nA reasonable structure:\n\n- Base component (weight 0.5): the Fear \u0026 Greed value, rescaled to −1..+1.\n- Market internals (weight 0.3): breadth-style measures, e.g. the fraction of your universe above a moving average, mapped to −1..+1.\n- Trend/volatility regime (weight 0.2): realized volatility percentile inverted, so calm regimes score positive.\n\n

python\nfrom dataclasses import dataclass\n\ndef rescale(value: float, lo: float, hi: float) -\u003e float:\n \"\"\"Map [lo, hi] to [-1, 1], clamped.\"\"\"\n x = (value - lo) / (hi - lo)\n return max(-1.0, min(1.0, 2 * x - 1))\n\n@dataclass\nclass SentimentScore:\n composite: float # -1 .. +1\n fng: float\n internals: float\n regime: float\n label: str\n\ndef compute_composite(fng_value: float | None,\n breadth: float | None,\n vol_pct: float | None) -\u003e SentimentScore:\n fng = rescale(fng_value, 0, 100) if fng_value is not None else 0.0\n internals = rescale(breadth, 0.0, 1.0) if breadth is not None else 0.0\n # High vol percentile -\u003e negative regime score.\n regime = rescale(1.0 - vol_pct, 0.0, 1.0) if vol_pct is not None else 0.0\n\n composite = 0.5 * fng + 0.3 * internals + 0.2 * regime\n\n if composite \u003e= 0.6:\n label = \"extreme_greed\"\n elif composite \u003c= -0.6:\n label = \"extreme_fear\"\n else:\n label = \"neutral\"\n\n return SentimentScore(composite, fng, internals, regime, label)\n

\n\nTwo properties make this worth the extra code. First, missing inputs collapse to neutral rather than skewing the score. Second, when the composite disagrees with your gut, you can print the three components and see which one moved. A single opaque number gives you nothing to debug.\n\nThe thresholds (±0.6) are arbitrary starting points. They should be tuned against your own trade history, not borrowed from a blog post — including this one.\n\n## The morning summary job\n\nThe scheduled job does the fetch, the composite, and the push. Keep it dumb: no order placement, no strategy state mutation. It writes a snapshot to disk and sends a message.\n\n

python\ndef morning_job() -\u003e None:\n fng = fetch_fng()\n breadth = load_breadth() # your own internals\n vol_pct = load_vol_percentile() # your own regime calc\n\n score = compute_composite(\n fng[\"value\"] if fng else None, breadth, vol_pct\n )\n\n snapshot = {\n \"date\": datetime.now(timezone.utc).date().isoformat(),\n \"fng_value\": fng[\"value\"] if fng else None,\n \"fng_ts\": fng[\"ts\"] if fng else None,\n \"composite\": round(score.composite, 3),\n \"label\": score.label,\n \"components\": {\n \"fng\": round(score.fng, 3),\n \"internals\": round(score.internals, 3),\n \"regime\": round(score.regime, 3),\n },\n }\n append_snapshot(snapshot) # one JSON line per day\n push_summary(snapshot) # Slack, Telegram, email — your call\n

\n\nappend_snapshot should write JSON lines to a dated file. That file becomes the historical series you'll join against trades later. If you skip this step and only push the summary to chat, you've thrown away the dataset you need to evaluate the filter.\n\n## The filter as a soft constraint\n\nThe filter's job is narrow: block new entries in the crowded direction. In extreme greed, suppress new long entries. In extreme fear, suppress new shorts. Exits, stops, and position sizing are untouched — you never want sentiment to trap you in a losing position.\n\nWire it in as a gate in the signal path, not inside the strategy logic:\n\n

python\ndef allow_new_entry(side: str, score: SentimentScore) -\u003e bool:\n if side == \"long\" and score.label == \"extreme_greed\":\n return False\n if side == \"short\" and score.label == \"extreme_fear\":\n return False\n return True\n

\n\n\"Soft\" is the key word. A hard filter that flips entire strategies off based on a daily number will produce long flat periods that are hard to attribute. A gate on new entries only is easier to reason about and easier to A/B.\n\nIf you want a middle ground, scale size instead of blocking:\n\n

python\ndef size_multiplier(score: SentimentScore) -\u003e float:\n if score.label in (\"extreme_greed\", \"extreme_fear\"):\n return 0.5\n return 1.0\n

\n\nBoth approaches are testable. Pick one, don't stack them, and log which one was active.\n\n## Logging sentiment with every trade\n\nThis is the part people skip, and it's the only part that decides whether the filter survives. Every trade record needs the sentiment state that was in force when the entry was decided.\n\n

python\ndef log_trade(trade: dict, score: SentimentScore) -\u003e None:\n record = {\n **trade,\n \"sentiment_composite\": round(score.composite, 3),\n \"sentiment_label\": score.label,\n \"sentiment_fng\": score.fng,\n \"sentiment_internals\": score.internals,\n \"sentiment_regime\": score.regime,\n }\n with open(\"data/trades.jsonl\", \"a\") as f:\n f.write(json.dumps(record) + \"\\n\")\n

\n\nLog the components, not just the label. When you later ask \"did the filter help?\", you'll want to slice by which component was driving the extreme, because the index component and the internals component may behave very differently.\n\nOnce trades carry the sentiment snapshot, evaluation is a groupby:\n\n

python\nimport pandas as pd\n\ndef evaluate_filter(path: str = \"data/trades.jsonl\") -\u003e pd.DataFrame:\n df = pd.read_json(path, lines=True)\n df[\"pnl\"] = df[\"exit_price\"] / df[\"entry_price\"] - 1\n df.loc[df[\"side\"] == \"short\", \"pnl\"] *= -1\n\n gated = df[\"sentiment_label\"].isin([\"extreme_greed\", \"extreme_fear\"])\n blocked = gated \u0026 (\n ((df[\"side\"] == \"long\") \u0026 (df[\"sentiment_label\"] == \"extreme_greed\")) |\n ((df[\"side\"] == \"short\") \u0026 (df[\"sentiment_label\"] == \"extreme_fear\"))\n )\n\n return df.groupby(\"sentiment_label\").agg(\n trades=(\"pnl\", \"size\"),\n avg_pnl=(\"pnl\", \"mean\"),\n hit_rate=(\"pnl\", lambda s: (s \u003e 0).mean()),\n blocked_by_filter=(\"pnl\", lambda s: blocked.loc[s.index].sum()),\n )\n

\n\nThe question you're answering is concrete: among trades taken in extreme greed, what was the average PnL of longs versus shorts? If longs in extreme greed performed the same as longs in neutral, the filter blocked trades for nothing. If they underperformed, it earned its place. Same logic for shorts in extreme fear.\n\nWatch the sample size. Extreme readings cluster — you may get twenty relevant trades in a year. Treat any conclusion from that with suspicion until you have more, or pool across related markets.\n\n## Demoting a filter when the data disagrees\n\nFilters accumulate. Someone adds a sentiment gate, someone else adds a volatility gate, and two years later nobody can say which one contributes. The discipline is to give every filter a review date and a demotion path.\n\nA simple workflow:\n\n1. Quarterly review. Re-run evaluate_filter with the latest trades. Compare gated versus ungated cohorts.\n2. Define the kill criterion up front. Example: \"If the blocked cohort's average PnL is within ±10 bps of the allowed cohort over at least 50 trades, demote to a size multiplier.\"\n3. Demote, don't delete. Move the rule from a hard gate to a size multiplier, or from a size multiplier to a logged-only annotation. Keep the logging running. You may promote it again in a different regime.\n4. Record the decision. A one-line note in the repo — date, criterion, outcome — saves the next person from re-litigating it.\n\nThe demotion step is where most systems rot. A filter that no longer earns its place but stays in the code silently shrinks your opportunity set. Keeping the sentiment fields in the trade log means demotion costs you nothing: the data keeps accumulating while the constraint loosens.\n\nOne more habit worth building: log the sentiment snapshot even when the filter is in \"annotation only\" mode. The cheapest way to evaluate a hypothetical filter is to run it in shadow for a quarter and compare what it would have blocked against what actually happened.\n\n## Wrapping up\n\nThe Fear \u0026 Greed index is a decent input and a poor signal. Used as a context layer — cached, composited with internals, applied as a soft gate on new entries, and logged on every trade — it becomes something you can measure. The measurement is the point. A filter that can't be evaluated is just an opinion with a cron job attached.\n\nMore notes like this ship every week on this site.\n\n---\n\n## Daily Picks\n\n\u003e The following pairs are selected from the multi-timeframe trend scanner (Gate.io futures) and are for technical-analysis study only — not investment advice.\n\u003e Data updated: 2026-11-04 08:46:36\n\n### Long\n\n| Pair | Signal | Price | Take Profit | Stop Loss | R/R |\n|------|--------|-------|-------------|-----------|-----|\n| AMD | 高质量标的做多 | $507.55 | $528.8671 | $496.8915 | 1:2.0 |\n\n_1 picks selected. Scanner runs every 15 minutes.
","positive_reactions_count":0,"cover_image":null,"tag_list":["fearandgreed","sentiment","filter","datapipeline"],"canonical_url":"https://www.hkcode.dpdns.org/en/posts/37.html","reading_time_minutes":7,"user":{"name":"mattleeee","username":"mattleeee","twitter_username":null,"github_username":"mattleeee","user_id":4160029,"website_url":null,"profile_image":"https://media2.dev.to/dynamic/image/width=640,height=640,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png","profile_image_90":"https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png"}},{"type_of":"article","id":4987027,"title":"Reading Paper-Trading Statistics: Win Rate, Profit Factor and Expectancy","description":"Why the Monthly Review Is the Only Review That Matters A paper-trading log accumulates...","published":true,"published_at":"2026-11-06T08:29:02.349Z","slug":"reading-paper-trading-statistics-win-rate-profit-factor-and-expectancy-a1k","path":"/mattleeee/reading-paper-trading-statistics-win-rate-profit-factor-and-expectancy-a1k","url":"https://dev.to/mattleeee/reading-paper-trading-statistics-win-rate-profit-factor-and-expectancy-a1k","comments_count":0,"public_reactions_count":0,"page_views_count":20,"published_timestamp":"2026-11-06T08:29:02Z","body_markdown":"## Why the Monthly Review Is the Only Review That Matters\n\nA paper-trading log accumulates trades faster than anyone reviews them. By the end of a month you have a trades.json with a few hundred entries and a vague feeling that \"it's working.\" That feeling is not a statistic. The monthly review exists to replace it with a table of numbers that can be wrong in specific, correctable ways.\n\nThis post covers the exact formulas for the core metrics, why they disagree with each other, how fees and funding quietly rewrite the story, and a Python function that turns a raw trade log into a monthly table you can push to WeChat.\n\n## The Trade Record\n\nEverything downstream depends on the shape of the input. A trade needs, at minimum:\n\n

python\n{\n \"id\": \"t_20240312_004\",\n \"ts_open\": \"2024-03-12T09:14:00Z\",\n \"ts_close\": \"2024-03-12T15:41:00Z\",\n \"strategy\": \"mom_breakout\",\n \"signal\": \"range_break\",\n \"side\": \"long\",\n \"qty\": 0.35,\n \"entry\": 41250.0,\n \"exit\": 41810.0,\n \"fee\": 4.12, # taker/maker fees, both legs, in quote currency\n \"funding\": -1.30, # net funding paid (negative) or received (positive)\n \"pnl_gross\": 196.0, # (exit - entry) * qty * direction\n \"pnl_net\": 192.58 # pnl_gross - fee + funding\n}\n

\n\nTwo rules matter here. First, pnl_net must be computed at write time and stored, not recomputed later from entry/exit — exchange fee tiers and funding schedules change, and you want the number that was true when the trade closed. Second, funding is signed: paid is negative, received is positive. Mixing the sign convention is the single most common source of a \"profitable\" paper account that isn't.\n\n## The Core Formulas\n\nGiven a list of closed trades, define winners as trades with pnl_net \u003e 0 and losers as pnl_net \u003c 0. Trades with exactly zero net PnL are excluded from win-rate denominators — they're neither wins nor losses, and counting them as losses flatters nothing and distorts everything.\n\n- Trade count: N = len(trades)\n- Win rate: W / (W + L) where W and L are counts of winners and losers.\n- Average win: mean(pnl_net for winners)\n- Average loss: mean(pnl_net for losers) — this is negative; report it as a negative number, not an absolute value, to avoid sign confusion downstream.\n- Expectancy (EV per trade): mean(pnl_net for all trades). Equivalently win_rate * avg_win + loss_rate * avg_loss.\n- Profit factor: sum(pnl_net for winners) / abs(sum(pnl_net for losers))\n- Max drawdown: computed on the cumulative net PnL curve, max(peak - equity) over time.\n- Longest losing streak: the longest consecutive run of pnl_net \u003c 0 in close-time order.\n\n

python\nimport statistics as st\n\ndef core_metrics(trades):\n if not trades:\n return {}\n pnls = [t[\"pnl_net\"] for t in trades]\n wins = [p for p in pnls if p \u003e 0]\n losses = [p for p in pnls if p \u003c 0]\n\n gross_win = sum(wins)\n gross_loss = abs(sum(losses))\n\n equity, peak, max_dd = 0.0, 0.0, 0.0\n streak = longest = 0\n for p in pnls:\n equity += p\n peak = max(peak, equity)\n max_dd = max(max_dd, peak - equity)\n streak = streak + 1 if p \u003c 0 else 0\n longest = max(longest, streak)\n\n return {\n \"n\": len(pnls),\n \"win_rate\": len(wins) / (len(wins) + len(losses)) if wins or losses else 0.0,\n \"avg_win\": st.mean(wins) if wins else 0.0,\n \"avg_loss\": st.mean(losses) if losses else 0.0,\n \"expectancy\": st.mean(pnls),\n \"profit_factor\": gross_win / gross_loss if gross_loss else float(\"inf\"),\n \"max_drawdown\": max_dd,\n \"longest_loss_streak\": longest,\n }\n

\n\nNote that drawdown here is in quote currency, not percent. Percent drawdown requires an account equity baseline, which a trade log alone doesn't give you. If you track a starting balance, divide by the running peak equity instead — but be explicit about which one you're reporting, because a 400-unit drawdown means very different things on a 2,000 and a 20,000 account.\n\n## Why Win Rate, Expectancy and Profit Factor Disagree\n\nThese three numbers answer different questions, and a system can look good on one while being bad on another.\n\n*High win rate, negative expectancy.* A strategy that wins 80% of the time but loses 5x as much on the losers has positive win rate and negative EV. Expectancy catches this; win rate hides it.\n\n*Profit factor and expectancy can point in opposite directions across sample sizes.* Expectancy is per trade, so it scales with trade count in aggregate but stays comparable between strategies with different frequencies. Profit factor is a ratio of sums, so a low-frequency strategy with one huge winner can post a profit factor of 3.0 on ten trades while its per-trade expectancy is mediocre and its variance is enormous. When comparing strategies, read expectancy for \"is this worth doing per trade\" and profit factor for \"how much cushion do the winners give the losers.\" A profit factor below 1.0 with positive expectancy is possible only when fees dominate — which is a signal that something in the cost model is wrong.\n\n*The honest combination* is expectancy \u003e 0, profit factor comfortably above 1.0 (1.2–1.5 is a reasonable working range for intraday), and a max drawdown you can survive. Any single one in isolation is a vanity metric.\n\n## Fees and Funding: Mark Them or Lie to Yourself\n\nThe most common paper-trading failure is computing PnL from entry and exit prices only. On a strategy with a 0.3% average edge per trade and 0.08% round-trip taker fees, fees eat over a quarter of the edge. Add funding on a position held overnight and a marginal strategy flips negative.\n\nTwo practices keep this honest:\n\n1. Store gross and net separately. Then run the review on net. Keep gross around only to measure cost drag: total_fees / gross_win tells you what fraction of your winners you're handing to the exchange.\n2. Attribute funding per trade. Funding is charged at discrete timestamps; if a trade spans several funding windows, sum the actual charges, don't estimate from average rates.\n\n

python\ndef cost_drag(trades):\n gross = sum(t[\"pnl_gross\"] for t in trades)\n fees = sum(t[\"fee\"] for t in trades)\n funding = sum(-t[\"funding\"] for t in trades) # cost is positive\n return {\n \"gross_pnl\": gross,\n \"fees\": fees,\n \"funding_cost\": funding,\n \"net_pnl\": gross - fees - funding,\n \"fee_share_of_gross\": fees / gross if gross \u003e 0 else None,\n }\n

\n\nIf fee_share_of_gross is above roughly 0.3 for an intraday system, the review should say so explicitly. That number is usually more actionable than the win rate.\n\n## Breaking Down per Strategy and per Signal\n\nAggregate numbers hide which component is carrying the account and which one is bleeding. The same metrics computed per strategy and per signal are the actual output of the review.\n\n

python\nfrom collections import defaultdict\n\ndef group_metrics(trades, key):\n buckets = defaultdict(list)\n for t in trades:\n buckets[t[key]].append(t)\n return {k: core_metrics(v) for k, v in buckets.items()}\n

\n\nRead the per-strategy table with sample size in view. A strategy with 12 trades and a 1.9 profit factor is noise; a strategy with 180 trades and a 1.1 profit factor is a real, thin edge. Require a minimum count (30 is a common floor) before drawing conclusions, and flag anything below it as \"insufficient sample\" in the output rather than ranking it.\n\n## The Monthly Table\n\nThe review itself is a filter by close timestamp, then the metrics, then a formatted table.\n\n

python\nfrom datetime import datetime, timezone\n\ndef monthly_review(trades, year, month):\n start = datetime(year, month, 1, tzinfo=timezone.utc)\n end = datetime(year + (month == 12), month % 12 + 1, 1, tzinfo=timezone.utc)\n window = [\n t for t in trades\n if start \u003c= datetime.fromisoformat(t[\"ts_close\"].replace(\"Z\", \"+00:00\")) \u003c end\n ]\n window.sort(key=lambda t: t[\"ts_close\"])\n\n return {\n \"period\": f\"{year}-{month:02d}\",\n \"overall\": core_metrics(window),\n \"by_strategy\": group_metrics(window, \"strategy\"),\n \"by_signal\": group_metrics(window, \"signal\"),\n \"costs\": cost_drag(window),\n }\n

\n\n## Pushing to WeChat\n\nWeChat Work (企业微信) group bots accept a JSON POST with a markdown content field. Format the table as markdown text and send it.\n\n

python\nimport json, urllib.request\n\ndef render_markdown(review):\n o = review[\"overall\"]\n c = review[\"costs\"]\n lines = [\n f\"**Paper Review {review['period']}**\",\n f\"Trades: {o['n']} | Win rate: {o['win_rate']:.1%}\",\n f\"Avg win: {o['avg_win']:.2f} | Avg loss: {o['avg_loss']:.2f}\",\n f\"Expectancy: {o['expectancy']:.2f}/trade | PF: {o['profit_factor']:.2f}\",\n f\"Max DD: {o['max_drawdown']:.2f} | Longest loss streak: {o['longest_loss_streak']}\",\n f\"Fees: {c['fees']:.2f} | Funding: {c['funding_cost']:.2f}\",\n \"\",\n \"**By strategy**\",\n ]\n for name, m in review[\"by_strategy\"].items():\n flag = \"\" if m[\"n\"] \u003e= 30 else \" (low n)\"\n lines.append(\n f\"- {name}: n={m['n']}{flag} EV={m['expectancy']:.2f} \"\n f\"PF={m['profit_factor']:.2f} WR={m['win_rate']:.1%}\"\n )\n return \"\\n\".join(lines)\n\ndef push_wechat(webhook_url, content):\n payload = json.dumps({\"msgtype\": \"markdown\", \"markdown\": {\"content\": content}}).encode()\n req = urllib.request.Request(\n webhook_url, data=payload, headers={\"Content-Type\": \"application/json\"}\n )\n with urllib.request.urlopen(req, timeout=10) as resp:\n return resp.read().decode()\n

\n\nWire it to a monthly cron job, load trades.json, run monthly_review, render, push. The whole pipeline is under a hundred lines and runs in a second.\n\n## What to Actually Do With the Table\n\nThe review is not a scoreboard; it's a triage list. Sort by expectancy per strategy. Anything with a negative expectancy and a sample of 30+ trades is a candidate for shutdown or a parameter change. Anything with a positive expectancy but a fee share above 0.3 goes to the execution side — maker orders, wider targets — before the signal side. Anything below 30 trades gets more data, not more tuning. The longest losing streak tells you what the equity curve will feel like; if it's longer than your patience, the system is right and the sizing is wrong.\n\nMore notes like this ship every week on this site.\n\n---\n\n## Daily Picks\n\n\u003e The following pairs are selected from the multi-timeframe trend scanner (Gate.io futures) and are for technical-analysis study only — not investment advice.\n\u003e Data updated: 2026-11-03 08:46:50\n\n### Short\n\n| Pair | Signal | Price | Take Profit | Stop Loss | R/R |\n|------|--------|-------|-------------|-----------|-----|\n| ESPORTS | 庄家做空 | $0.0147 | $0.0139 | $0.0153 | 1:1.6 |\n\n_1 picks selected. Scanner runs every 15 minutes.","positive_reactions_count":0,"cover_image":null,"tag_list":["expectancy","profitfactor","winrate","monthlyreview"],"canonical_url":"https://www.hkcode.dpdns.org/en/posts/36.html","reading_time_minutes":7,"user":{"name":"mattleeee","username":"mattleeee","twitter_username":null,"github_username":"mattleeee","user_id":4160029,"website_url":null,"profile_image":"https://media2.dev.to/dynamic/image/width=640,height=640,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png","profile_image_90":"https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png"}},{"type_of":"article","id":4987024,"title":"Weak-Signal Blacklisting: Two-Month Windows and a 70% Win-Rate Gate","description":"The Problem with \"Just Skip the Weak Ones\" Every signal generator eventually produces...","published":true,"published_at":"2026-11-06T08:28:26.095Z","slug":"weak-signal-blacklisting-two-month-windows-and-a-70-win-rate-gate-12kj","path":"/mattleeee/weak-signal-blacklisting-two-month-windows-and-a-70-win-rate-gate-12kj","url":"https://dev.to/mattleeee/weak-signal-blacklisting-two-month-windows-and-a-70-win-rate-gate-12kj","comments_count":0,"public_reactions_count":0,"page_views_count":10,"published_timestamp":"2026-11-06T08:28:26Z","body_markdown":"## The Problem with \"Just Skip the Weak Ones\"\n\nEvery signal generator eventually produces marginal setups. The breakout that barely clears the range, the mean-reversion entry where the z-score is 1.8 instead of 2.0, the momentum flip that fires on a thin bar. Traders eyeball these and skip them, but eyeballing doesn't scale and doesn't leave an audit trail. You need a rule: if a signal type can't clear a win-rate bar over a recent window, it goes on probation, and if it keeps failing, it gets blacklisted until it earns its way back.\n\nThis post describes a small, self-contained implementation of that policy: a two-month rolling window per signal type, a 70% win-rate gate, and a three-strike blacklist. The interesting parts are the bookkeeping details — how you key the statistics, how you persist them safely, and what exactly you're blacklisting.\n\n## Defining a Signal Type Key\n\nThe whole system collapses if your aggregation key is wrong. A \"signal type\" is not the instrument, not the timestamp, and not the direction. It's the rule identity: the thing whose historical win rate you're actually trying to estimate.\n\nA good key is a stable, human-readable string composed of the components that define the rule:\n\n

python\nfrom dataclasses import dataclass\n\n@dataclass(frozen=True)\nclass SignalType:\n family: str # e.g. \"range_breakout\"\n timeframe: str # e.g. \"1h\"\n variant: str # e.g. \"atr_buffer_0.5\"\n\n def key(self) -\u003e str:\n return f\"{self.family}|{self.timeframe}|{self.variant}\"\n

\n\nThree rules for the key:\n\n1. Include every parameter that can change the edge. If atr_buffer_0.5 and atr_buffer_1.0 behave differently, they're different signal types. Aggregating them hides a weak variant behind a strong one.\n2. Exclude the instrument. The same breakout rule applied to ten instruments is still one signal type — you're testing the rule's edge, not its luck on one symbol. (More on this distinction below.)\n3. Never derive the key from runtime state. No timestamps, no trade IDs, no randomly generated suffixes. If the key changes between runs, your history silently resets.\n\nIf you version rules, put the version in the key: range_breakout|1h|atr_buffer_0.5|v3. A rule change is a new signal type with no history — which is exactly the conservative behavior you want.\n\n## Rolling-Window Bookkeeping\n\nYou don't need a database for this. A JSON file per signal type, or one file with a dict keyed by signal type, is enough — as long as writes are atomic and the window is pruned on every update.\n\nEach record stores the outcome of one closed paper trade and the timestamp of its close:\n\n

python\nimport json\nimport os\nimport tempfile\nfrom datetime import datetime, timedelta, timezone\nfrom pathlib import Path\n\nWINDOW_DAYS = 60 # ~two months\nWIN_RATE_GATE = 0.70 # required to stay tradeable\nMIN_SAMPLES = 10 # don't judge on 3 trades\nMAX_CONSECUTIVE_FAILURES = 3\n\nSTORE_PATH = Path(\"signal_stats.json\")\n\ndef _atomic_write(path: Path, data: dict) -\u003e None:\n fd, tmp = tempfile.mkstemp(dir=path.parent, suffix=\".tmp\")\n try:\n with os.fdopen(fd, \"w\") as f:\n json.dump(data, f, indent=2, sort_keys=True)\n f.flush()\n os.fsync(f.fileno())\n os.replace(tmp, path) # atomic on POSIX and Windows\n except BaseException:\n os.unlink(tmp)\n raise\n\ndef load_stats() -\u003e dict:\n if not STORE_PATH.exists():\n return {}\n with STORE_PATH.open() as f:\n return json.load(f)\n\ndef save_stats(stats: dict) -\u003e None:\n _atomic_write(STORE_PATH, stats)\n

\n\nThe os.replace call is the atomicity guarantee: a crash mid-write leaves either the old file or the new one, never a truncated JSON blob. On Windows, os.replace overwrites the destination, which os.rename does not — that's why it's used here.\n\nNow the update path. Every time a paper trade closes, append it, prune anything older than the window, and recompute the gate:\n\n

python\ndef record_outcome(signal_key: str, won: bool, closed_at: datetime | None = None) -\u003e dict:\n closed_at = closed_at or datetime.now(timezone.utc)\n stats = load_stats()\n entry = stats.setdefault(signal_key, {\n \"trades\": [],\n \"consecutive_failures\": 0,\n \"blacklisted\": False,\n })\n\n entry[\"trades\"].append({\"closed_at\": closed_at.isoformat(), \"won\": won})\n\n cutoff = closed_at - timedelta(days=WINDOW_DAYS)\n entry[\"trades\"] = [\n t for t in entry[\"trades\"]\n if datetime.fromisoformat(t[\"closed_at\"]) \u003e= cutoff\n ]\n\n trades = entry[\"trades\"]\n n = len(trades)\n wins = sum(t[\"won\"] for t in trades)\n win_rate = wins / n if n else 0.0\n\n if n \u003c MIN_SAMPLES:\n entry[\"consecutive_failures\"] = 0\n entry[\"blacklisted\"] = False\n elif win_rate \u003e= WIN_RATE_GATE:\n entry[\"consecutive_failures\"] = 0\n entry[\"blacklisted\"] = False\n else:\n entry[\"consecutive_failures\"] += 1\n if entry[\"consecutive_failures\"] \u003e= MAX_CONSECUTIVE_FAILURES:\n entry[\"blacklisted\"] = True\n\n save_stats(stats)\n return entry\n

\n\nA few deliberate choices:\n\n- The failure counter increments per evaluation, not per trade. The gate is checked when a trade closes. Three consecutive evaluations below 70% (with enough samples) trigger the blacklist. This prevents a single bad stretch from flipping the switch instantly, while still reacting within a few closed trades.\n- MIN_SAMPLES prevents premature judgment. A signal type with four paper trades and three wins isn't a 75% winner; it's noise. Until the sample is large enough, the counter resets and nothing gets blacklisted.\n- Pruning happens on write, not on read. Reads stay cheap and consistent; the file never grows unbounded.\n\nThe gate check itself is trivial:\n\n

python\ndef is_tradeable(signal_key: str) -\u003e bool:\n entry = load_stats().get(signal_key)\n if entry is None:\n return True # no history: allow, but it's on probation by default\n return not entry[\"blacklisted\"]\n

\n\nNote the default: an unknown signal type is tradeable. That's fine for paper trading, where you want new signal types to accumulate history. If you're routing live capital, invert the default and require explicit qualification.\n\n## Blacklisting a Signal vs. Blacklisting an Instrument\n\nThese are different mechanisms and conflating them causes real damage.\n\n*Blacklisting a signal type* says: \"This rule, in its current form, doesn't clear the bar.\" The rule stops generating tradeable entries everywhere. The underlying instrument is untouched.\n\n*Blacklisting an instrument* says: \"This symbol is untradeable right now\" — because of liquidity, spread, a corporate action, or a data quality problem. The signal rule is untouched; it just can't fire on that symbol.\n\nKeep them in separate stores with separate lifecycles:\n\n

python\ndef is_tradeable(signal_key: str, instrument: str) -\u003e bool:\n stats = load_stats()\n if stats.get(signal_key, {}).get(\"blacklisted\", False):\n return False\n instrument_flags = load_instrument_flags()\n if instrument_flags.get(instrument, {}).get(\"blocked\", False):\n return False\n return True\n

\n\nWhy the separation matters: if you blacklist the signal because it performed poorly on one illiquid instrument, you'll wrongly shut down a rule that works fine elsewhere. Conversely, if you blacklist the instrument because one rule struggles on it, you'll block every other rule that trades it profitably. The statistics only mean something if the key matches the scope of the claim. A win rate is a property of a rule; liquidity is a property of a symbol. Don't mix them.\n\n## Start Probation on Paper, Not Live Money\n\nThe three-strike counter is a blunt instrument. It fires after a handful of bad evaluations and can fire on a signal type that was genuinely profitable last quarter but hit a regime shift. You do not want that discovery to happen with real positions.\n\nSo the lifecycle is:\n\n1. New signal type → paper trading. It accumulates outcomes in the rolling window. No capital at risk while the sample is small.\n2. Paper history clears the gate → eligible for live routing. The gate decision uses the same window and the same 70% threshold.\n3. Live performance degrades → back to paper. When the live win rate over the window drops below the gate for three consecutive evaluations, the signal type is blacklisted and demoted. It re-qualifies only by rebuilding a paper record that clears the gate again.\n\nThe re-qualification step deserves a note. When a blacklisted signal type re-enters paper trading, you have a choice: keep its old trades in the window, or reset the history. Resetting is usually correct — the old trades are what got it blacklisted, and leaving them in the window means the win rate can't recover until they age out (up to two months of dead time). A clean reset gives the rule a fair fresh evaluation:\n\n

python\ndef requalify(signal_key: str) -\u003e None:\n stats = load_stats()\n entry = stats.get(signal_key)\n if entry is None:\n return\n entry[\"trades\"] = []\n entry[\"consecutive_failures\"] = 0\n entry[\"blacklisted\"] = False\n save_stats(stats)\n

\n\nCall this only when you deliberately move a signal type back to paper. Don't automate it — a human should confirm that the rule was actually revised or that the market conditions that caused the failure have plausibly changed.\n\n## Operational Notes\n\n- Timezones. Store all timestamps in UTC and parse with datetime.fromisoformat. Mixing local and UTC timestamps in a rolling window produces subtly wrong prunes that are painful to debug.\n- Concurrency. If multiple processes write the same JSON file, os.replace prevents corruption but not lost updates. For anything beyond a single trading loop, wrap the read-modify-write in a file lock (msvcrt.locking on Windows, fcntl.flock on POSIX) or move to SQLite.\n- Audit trail. Keep a separate append-only log of every outcome and every blacklist transition. The JSON store is working state; the log is what lets you reconstruct why a signal type was shut off six weeks ago.\n- Window length. Sixty days is a starting point, not a law. Shorter windows react faster but blacklist on noise; longer windows are stable but slow to catch decay. The two-month default balances the two, and the gate threshold should be tuned against your own paper results, not copied blindly.\n\nThe design is intentionally boring: one key, one window, one threshold, three strikes. Boring bookkeeping is what makes the gate trustworthy — and a gate you don't trust is just a number in a config file.\n\nMore notes like this ship every week on this site.\n\n---\n\n## Daily Picks\n\n\u003e The following pairs are selected from the multi-timeframe trend scanner (Gate.io futures) and are for technical-analysis study only — not investment advice.\n\u003e Data updated: 2026-11-02 08:46:42\n\n### Long\n\n| Pair | Signal | Price | Take Profit | Stop Loss | R/R |\n|------|--------|-------|-------------|-----------|-----|\n| AIN | 高质量标的做多 | $0.2872 | $0.3061 | $0.2777 | 1:2.0 |\n\n_1 picks selected. Scanner runs every 15 minutes.
","positive_reactions_count":0,"cover_image":null,"tag_list":["weaksignals","blacklist","winrategate","rollingwindow"],"canonical_url":"https://www.hkcode.dpdns.org/en/posts/35.html","reading_time_minutes":7,"user":{"name":"mattleeee","username":"mattleeee","twitter_username":null,"github_username":"mattleeee","user_id":4160029,"website_url":null,"profile_image":"https://media2.dev.to/dynamic/image/width=640,height=640,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png","profile_image_90":"https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png"}},{"type_of":"article","id":4955307,"title":"Adaptive Stop-Loss Thresholds Instead of Hardcoded Percentages","description":"A fixed -3% stop does not mean the same thing on every instrument. On a volatile altcoin, 3% is...","published":true,"published_at":"2026-11-01T01:01:48.511Z","slug":"adaptive-stop-loss-thresholds-instead-of-hardcoded-percentages-300a","path":"/mattleeee/adaptive-stop-loss-thresholds-instead-of-hardcoded-percentages-300a","url":"https://dev.to/mattleeee/adaptive-stop-loss-thresholds-instead-of-hardcoded-percentages-300a","comments_count":0,"public_reactions_count":0,"page_views_count":30,"published_timestamp":"2026-11-01T01:01:48Z","body_markdown":"A fixed -3% stop does not mean the same thing on every instrument. On a volatile altcoin, 3% is inside the normal noise band, so the stop fires on routine wiggles and the strategy never gets to express its edge. On a low-volatility index product, 3% is a wide move, so the stop sits far beyond where the strategy's own logic would have exited. The same number produces opposite behaviour depending on the thing it's attached to.\n\nThe fix is not to tune the percentage per instrument by hand. It's to derive the safety floor from the strategy's own stop calculation, so the floor protects capital without overriding the strategy's exit logic. The formula used here:\n\n

python\nthreshold = -max(3.0, abs(strategy_stop_pct) * 2.0)\n

\n\nRead it as two clauses. The abs(strategy_stop_pct) * 2.0 clause says: never force-close before the strategy's own stop has had a chance to work, and only intervene once the position has drifted to roughly twice the intended stop distance. The max(3.0, ...) clause says: there is always at least a 3% floor, so a strategy with a very tight stop (say 0.5%) doesn't get a 1% force-close that would fire on spread and slippage alone.\n\n## What each clause is protecting\n\nThe * 2.0 multiplier is the interesting part. A strategy that stops at 2% has a defined risk budget. If the price is at -2.5%, the strategy's stop should have already acted; if it hasn't, something is wrong (stale data, a gap, a broker reject). Waiting until -4% gives the strategy room to do its job and cuts the position only when the loss is clearly beyond what the strategy signed up for. That's the \"respect the strategy\" half.\n\nThe max(3.0, ...) half is the \"protect capital\" floor. Without it, a strategy with a 0.5% stop yields a 1% force-close threshold, which is noise-level for most liquid instruments. The force-close would become the actual stop, and the strategy's own logic would be irrelevant. The 3% minimum guarantees the force-close is a genuine backstop, not a second, tighter stop.\n\nThe result is a threshold that scales with the instrument's volatility as the strategy sees it, because strategy_stop_pct is itself volatility-derived. That's the whole point: the floor inherits the strategy's view of normal movement instead of imposing an external one.\n\n## Computing strategy_stop_pct per instrument\n\nstrategy_stop_pct must come from the strategy's own stop logic, not from a separate volatility estimate. If the strategy uses ATR-based stops, use that. If it uses a percentage of price derived from recent range, use that. The floor and the strategy must agree on what \"the stop\" means, or the * 2.0 multiplier is meaningless.\n\nA common pattern is an ATR stop with a multiplier:\n\n

python\nimport pandas as pd\n\ndef atr(df: pd.DataFrame, period: int = 14) -\u003e pd.Series:\n high, low, close = df[\"high\"], df[\"low\"], df[\"close\"]\n prev_close = close.shift(1)\n tr = pd.concat([\n high - low,\n (high - prev_close).abs(),\n (low - prev_close).abs(),\n ], axis=1).max(axis=1)\n return tr.ewm(alpha=1 / period, adjust=False).mean()\n\ndef strategy_stop_pct(df: pd.DataFrame, atr_mult: float = 2.0,\n atr_period: int = 14) -\u003e pd.Series:\n \"\"\"Stop distance as a fraction of current price.\"\"\"\n a = atr(df, atr_period)\n return (atr_mult * a) / df[\"close\"]\n

\n\nThen the floor follows directly:\n\n

python\ndef safety_floor_pct(stop_pct: pd.Series,\n min_floor: float = 3.0,\n mult: float = 2.0) -\u003e pd.Series:\n \"\"\"Returns positive magnitude; apply as -value when comparing PnL.\"\"\"\n return (stop_pct.abs() * mult).clip(lower=min_floor)\n

\n\nTwo details matter here.\n\nFirst, stop_pct is a series, not a scalar. ATR changes, so the stop distance changes, so the floor changes. Recompute it on every bar or every position update. A floor computed once at entry will be wrong by the time the position has moved.\n\nSecond, clip(lower=min_floor) implements the max() cleanly and keeps the output vectorised. For a strategy with a 2% stop, the floor is 4%. For a 0.8% stop, the floor is 3% (the minimum wins). For a 6% stop, the floor is 12%.\n\nOne caveat on the ATR version: ATR is in price units, so dividing by close converts it to a fraction. For instruments with very different price scales (a $0.02 token vs. a $400 stock), the fraction is the only comparable quantity. Keep everything in fractional terms until the final PnL comparison.\n\n## Applying it to open positions\n\nThe check itself is a simple comparison against unrealised PnL as a fraction:\n\n

python\ndef should_force_close(entry_price: float,\n current_price: float,\n stop_pct_now: float,\n min_floor: float = 3.0,\n mult: float = 2.0) -\u003e bool:\n pnl_pct = (current_price - entry_price) / entry_price * 100.0\n floor = max(min_floor, abs(stop_pct_now) * mult)\n return pnl_pct \u003c= -floor\n

\n\nCall this on every tick or bar, passing the current stop_pct_now rather than the value at entry. If the instrument has calmed down since entry, the floor tightens; if volatility has expanded, the floor widens. That's the adaptive part, and it's what a hardcoded percentage can't do.\n\nFor a batch of positions, vectorise it:\n\n

python\ndef force_close_mask(entry: pd.Series, current: pd.Series,\n stop_pct_now: pd.Series,\n min_floor: float = 3.0,\n mult: float = 2.0) -\u003e pd.Series:\n pnl_pct = (current - entry) / entry * 100.0\n floor = (stop_pct_now.abs() * mult).clip(lower=min_floor)\n return pnl_pct \u003c= -floor\n

\n\n## Backtesting the floor's effect on surviving trades\n\nThe number that matters is not how many trades the floor cuts. It's how many trades the floor cuts that would have recovered, and how much tail risk it removes. A floor that never fires is doing nothing; a floor that fires on trades that would have hit their own stop anyway is redundant. The useful cases are the ones in between: trades that drifted past the strategy's intended stop without triggering it, and trades that recovered.\n\nTag each trade with whether the floor fired and whether the trade would have been profitable without it:\n\n

python\ndef evaluate_floor(trades: pd.DataFrame) -\u003e dict:\n \"\"\"\n trades columns: pnl_pct (final, without floor), floor_pnl_pct\n (PnL at the moment the floor would have fired, NaN if it never fired),\n would_recover (bool: final pnl_pct \u003e 0 without floor).\n \"\"\"\n fired = trades[\"floor_pnl_pct\"].notna()\n cut = trades[fired]\n return {\n \"n_trades\": len(trades),\n \"n_fired\": int(fired.sum()),\n \"fired_rate\": fired.mean(),\n \"avg_pnl_fired\": cut[\"floor_pnl_pct\"].mean(),\n \"avg_pnl_if_held\": cut[\"pnl_pct\"].mean(),\n \"recovered_after_cut\": int(cut[\"would_recover\"].sum()),\n \"recovered_pnl_cost\": cut.loc[cut[\"would_recover\"], \"pnl_pct\"].sum(),\n \"tail_saved\": (\n trades.loc[~fired, \"pnl_pct\"].nsmallest(20).mean()\n - cut[\"floor_pnl_pct\"].mean()\n if fired.any() else float(\"nan\")\n ),\n }\n

\n\nThe two numbers to read together are recovered_pnl_cost and the change in the worst-case tail. If the floor cuts a handful of trades that would have recovered for a small total PnL cost, and it removes a fat left tail, it's earning its keep. If recovered_pnl_cost is large relative to the tail it removes, the multiplier is too tight: raise mult from 2.0 toward 2.5 or 3.0 and re-run. If the floor barely fires, the multiplier is too loose or the strategies' stops are already doing the work.\n\nAlso break the results down by instrument. A floor that looks good in aggregate can be doing all its work on one volatile instrument and nothing elsewhere. If one instrument accounts for most of the fired trades, its strategy_stop_pct is probably mis-specified rather than the floor being wrong.\n\nWatch the sample size. Floor events are rare by construction, so a backtest with 200 trades may produce a dozen firings, which is not enough to conclude anything. Pool across instruments and across a long enough window that the volatility regime shifts at least once. If the floor's contribution only shows up in a single high-volatility month, you're measuring one event, not an edge.\n\n## When a hard floor must override the adaptive value\n\nThe adaptive threshold assumes strategy_stop_pct is meaningful. That assumption fails in specific, recognisable cases, and in those the hard floor must win outright.\n\n*Stale or missing volatility data.* If ATR is NaN, zero, or computed from a thin book, stop_pct collapses toward zero and the adaptive floor tightens to the 3% minimum. That's benign. But if stop_pct is stale — carried over from a prior regime — the floor can be far too wide. Detect it:\n\n

python\ndef effective_floor(stop_pct_now: float,\n stop_pct_prev: float,\n min_floor: float = 3.0,\n mult: float = 2.0,\n max_floor: float = 15.0) -\u003e float:\n if not (stop_pct_now \u003e 0):\n return min_floor # bad data: fall back to hard floor\n adaptive = max(min_floor, abs(stop_pct_now) * mult)\n if abs(stop_pct_now - stop_pct_prev) / stop_pct_prev \u003e 0.5:\n adaptive = max(adaptive, min_floor) # regime jump: don't trust the ratio\n return min(adaptive, max_floor) # hard ceiling regardless\n

\n\n*A hard ceiling.* The adaptive formula has no upper bound. A strategy with a 10% stop yields a 20% force-close threshold, which for most instruments is not risk management, it's capitulation. Cap it. The max_floor argument above enforces a ceiling that no adaptive value can exceed. The exact number depends on the instrument's typical drawdown distribution, but it should exist and it should be hardcoded, because its purpose is to override the adaptive logic when the adaptive logic produces something absurd.\n\n*Gap and liquidity events.* The floor is evaluated on observed prices. If an instrument gaps through the threshold overnight, the force-close executes at a worse price than the threshold. That's unavoidable, but it means the floor's effective loss can exceed its nominal value. In backtests, model the fill at the next available price, not at the threshold, or the floor will look more protective than it is.\n\n*Strategy stop wider than the ceiling.* If strategy_stop_pct * 2.0 exceeds max_floor, the ceiling wins and the strategy's stop is effectively overridden. That's the correct outcome: the strategy has asked for more risk than the account's policy allows, and the policy should win. Log these cases — they usually mean the strategy is mis-parameterised for that instrument, not that the floor is too tight.\n\n## The design principle\n\nThe floor exists to catch what the strategy's own stop missed: gaps, stale stops, broker rejects, logic errors. It should be the outer layer of defence, not the first line. Deriving it from strategy_stop_pct keeps it in the right place — wide enough that the strategy's exit logic runs first, tight enough that a runaway loss gets cut before it compounds. The hard floor and the hard ceiling bracket the adaptive value so that bad inputs can't push it into useless territory. Everything between those brackets is the strategy's own risk definition, scaled by a constant.\n\nMore notes like this ship every week on this site.","positive_reactions_count":0,"cover_image":null,"tag_list":["stoploss","adaptivethreshold","volatility","riskmanagement"],"canonical_url":"https://www.hkcode.dpdns.org/en/posts/34.html","reading_time_minutes":7,"user":{"name":"mattleeee","username":"mattleeee","twitter_username":null,"github_username":"mattleeee","user_id":4160029,"website_url":null,"profile_image":"https://media2.dev.to/dynamic/image/width=640,height=640,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png","profile_image_90":"https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png"}},{"type_of":"article","id":4950138,"title":"Partial Take-Profit: Bank Half at 3%, Let the Rest Run","description":"The Problem With a Single Take-Profit A fixed take-profit forces a binary choice. Set it...","published":true,"published_at":"2026-10-31T01:01:47.169Z","slug":"partial-take-profit-bank-half-at-3-let-the-rest-run-3fi","path":"/mattleeee/partial-take-profit-bank-half-at-3-let-the-rest-run-3fi","url":"https://dev.to/mattleeee/partial-take-profit-bank-half-at-3-let-the-rest-run-3fi","comments_count":0,"public_reactions_count":0,"page_views_count":10,"published_timestamp":"2026-10-31T01:01:47Z","body_markdown":"## The Problem With a Single Take-Profit\n\nA fixed take-profit forces a binary choice. Set it tight and you cut winners that would have kept running. Set it wide and you watch a 5% open profit evaporate back to zero because the target was never hit. Both outcomes come from the same design flaw: the exit is a single point, and the market gets one vote on whether you keep everything.\n\nPartial take-profit splits that vote. The version I run in the paper trader: when unrealized profit on the open position reaches 3%, sell half the position and move the stop on the remainder to break-even. The remaining half targets the original 6%. The first leg banks a guaranteed gain; the second leg keeps upside exposure at zero risk (modulo fees, which we'll get to).\n\nThe strategy logic is trivial. The bookkeeping is where implementations break, so most of this post is about state.\n\n## Position State\n\nEvery open position needs enough state to answer: how much is left, what did it cost, what's been realized, and has the partial already fired. A dataclass is the cleanest container:\n\n

python\nfrom dataclasses import dataclass, field\n\n@dataclass\nclass Position:\n entry_price: float\n qty: float\n stop_price: float\n target_price: float\n partial_level: float\n partial_fraction: float = 0.5\n partial_done: bool = False\n realized_pnl: float = 0.0\n fees_paid: float = 0.0\n initial_qty: float = field(init=False)\n\n def __post_init__(self):\n self.initial_qty = self.qty\n

\n\nTwo fields do the heavy lifting. partial_done is the latch that guarantees the 3% rule fires once, not on every candle that closes above the threshold. initial_qty preserves the original size so you can report percentage-of-position metrics after the exit.\n\nNote that entry_price never changes. After a partial exit, the remaining shares still carry the same cost basis — the average entry of the remaining position is unchanged. What changes is how much capital is at risk. If you recompute an \"average entry\" by blending realized and unrealized, you'll corrupt the R-multiple math later. Keep cost basis immutable; track realized PnL separately.\n\n## The Candle Loop and Check Order\n\nThe rule must be evaluated exactly once per candle, and the order of checks matters because a single candle can touch the stop, the partial level, and the target. Here's the loop I use:\n\n

python\ndef process_candle(pos: Position, candle, fee_rate: float = 0.001):\n o, h, l, c = candle.open, candle.high, candle.low, candle.close\n\n # 1. Gap handling: if the open is already beyond a level,\n # fill at the open, not at the level.\n # 2. Stop check first (conservative: assume worst case within candle).\n stop_hit = l \u003c= pos.stop_price\n target_hit = h \u003e= pos.target_price\n partial_hit = (not pos.partial_done) and h \u003e= pos.partial_level\n\n # Conservative ordering: stop before target when both touched.\n if stop_hit:\n fill = min(o, pos.stop_price) if o \u003c pos.stop_price else pos.stop_price\n close_remainder(pos, fill, fee_rate)\n return \"stopped\"\n\n if partial_hit:\n take_partial(pos, max(o, pos.partial_level), fee_rate)\n # Re-evaluate target on the same candle with reduced size.\n if h \u003e= pos.target_price:\n fill = max(o, pos.target_price)\n close_remainder(pos, fill, fee_rate)\n return \"partial+target\"\n return \"partial\"\n\n if target_hit:\n fill = max(o, pos.target_price)\n close_remainder(pos, fill, fee_rate)\n return \"target\"\n\n return \"open\"\n

\n\nThe ordering decisions, spelled out:\n\n*Stop before partial.* If a candle's range covers both the stop and the 3% level, assume the stop got hit first. Without tick data you cannot know the intrabar sequence, and assuming the favorable fill first is how backtests develop a personality disorder. Conservative ordering costs you some hypothetical profit and keeps the equity curve honest.\n\n*Partial before target.* Once the 3% level triggers, the target check runs against the reduced position. If one candle runs from below 3% straight through 6%, you take the partial at 3% and the remainder at 6%. That's the intended behavior: the rule is a schedule, not an either/or.\n\n*Gap opens.* If the candle opens above the target, the fill is the open, not the target — you don't get to sell at a price the market skipped. The max(o, level) / min(o, level) expressions handle long positions; flip them for shorts. Gap opens beyond both the partial and the target (open at 7%) mean the partial fills at the open too, and the remainder also fills at the open. Bank it and move on.\n\n### The Partial Fill Itself\n\n

python\ndef take_partial(pos: Position, fill_price: float, fee_rate: float):\n qty = pos.qty * pos.partial_fraction\n proceeds = qty * fill_price\n fee = proceeds * fee_rate\n cost = qty * pos.entry_price\n\n pos.realized_pnl += proceeds - fee - cost\n pos.fees_paid += fee\n pos.qty -= qty\n pos.partial_done = True\n\n # Move stop to break-even, adjusted for fees already sunk.\n pos.stop_price = pos.entry_price\n

\n\nTwo subtleties here. First, pos.qty -= qty uses the fractional quantity, so a 50% exit on an odd lot leaves a fractional remainder — fine for crypto, and for equities you'd round down and keep the residual. Second, the break-even stop is set to entry_price, not to entry_price + fees. That's the version most people write, and it's slightly wrong; see the fee section.\n\n## Edge Cases That Bite\n\n*The \"free ride\" isn't free.* Moving the stop to break-even means the remainder exits flat before fees. If you paid 0.1% on entry and 0.1% on exit, a break-even stop on the remainder loses roughly 0.2% of the remaining notional. Over hundreds of trades this shows up. The fix is to set the stop at the fee-adjusted break-even:\n\n

python\ndef breakeven_price(entry: float, entry_fee_rate: float,\n exit_fee_rate: float) -\u003e float:\n # Price at which net PnL on the remainder is zero.\n return entry * (1 + entry_fee_rate) / (1 - exit_fee_rate)\n

\n\nWith 0.1% each way, that's entry × 1.001 / 0.999 ≈ entry × 1.002. Two basis points above entry — small, but it converts \"free ride\" from a slogan into an arithmetic fact. Whether the fee-adjusted stop or the plain break-even stop performs better is an empirical question; test both.\n\n*Partial level below entry after slippage.* If your fill model includes slippage, the realized PnL on the partial can be negative even though the trigger price was 3% above entry. Guard the expectancy math against this by using actual fill prices everywhere, never the trigger levels.\n\n*Restart safety.* If the trader persists state between runs, partial_done must be serialized. Reconstructing a position from \"current qty and entry\" alone will re-fire the partial after a restart. Persist the full dataclass.\n\n*One partial per position.* The latch is not an optimization; it's the semantics. If you want multiple scale-out levels (3%, 5%, 7%), replace the boolean with a list of pending levels and pop them as they trigger. The bookkeeping pattern is identical.\n\n## How Partial TP Changes Expectancy\n\nThe standard expectancy formula for a single-target system is:\n\n

plaintext\nE = p_win * avg_win - p_loss * avg_loss\n

\n\nPartial exits break avg_win into two components and change the outcome distribution. With half out at 3% and half at 6% (ignoring fees and the break-even stop for a moment), a full winner returns 0.5 * 3% + 0.5 * 6% = 4.5% of position notional. A trade that reaches 3% but stalls returns 0.5 * 3% + 0 = 1.5% instead of riding back to zero. A trade stopped at break-even after the partial returns 1.5% instead of a full loss.\n\nThe trade-off: you've capped the average winner at 4.5% instead of 6%, in exchange for converting some would-be losers into small winners and shrinking the loss tail. Whether that improves expectancy depends on the distribution of maximum favorable excursion (MFE) — specifically, how many trades reach 3% versus 6%. If most trades that hit 3% also hit 6%, the partial just costs you half a percent per winner. If a large fraction stall between 3% and 6%, the partial is where the edge lives.\n\nYou can't reason your way to the answer; you measure it. Log MFE per trade and build the histogram:\n\n

python\nimport numpy as np\n\ndef mfe_histogram(trades, bins=(0, 1, 2, 3, 4, 5, 6, 8, 10)):\n mfe = np.array([t.max_favorable_pct for t in trades])\n hist, edges = np.histogram(mfe, bins=bins)\n for lo, hi, n in zip(edges[:-1], edges[1:], hist):\n print(f\"{lo:\u003e4.0f}%-{hi:\u003c4.0f}% {n:\u003e5d} {'#' * int(50 * n / max(hist))}\")\n

\n\nIf the 3–6% bucket is fat, the partial earns its keep. If it's nearly empty, a single wider target is simpler and better.\n\n### A Cleaner Expectancy Formula\n\nOnce partials exist, per-trade expectancy is best computed from realized cash flows rather than win/loss averages:\n\n

python\ndef expectancy(closed_trades):\n # Each trade: dict with 'pnl_pct' (net return on initial notional)\n pnl = np.array([t[\"pnl_pct\"] for t in closed_trades])\n return pnl.mean(), pnl.std() / np.sqrt(len(pnl))\n

\n\nBecause every trade now has a continuum of outcomes (full stop, partial + break-even, partial + target, direct target), the win/loss split loses information. Mean net return per trade on initial notional, with a standard error, is the honest summary. Report it alongside the MFE histogram and the fraction of trades where the partial fired.\n\n## What Actually Matters\n\nThe partial take-profit is a small piece of logic wrapped in a lot of state discipline. Get these right and the rest follows:\n\n- Latch the partial with an explicit flag and persist it.\n- Evaluate stop → partial → target, once per candle, with gap-aware fills.\n- Keep entry price immutable; track realized PnL separately.\n- Use fee-adjusted break-even if you care about the \"free\" part being real.\n- Judge the rule by the MFE distribution and net expectancy, not by how it feels.\n\nMore notes like this ship every week on this site.\n\n---\n\n## Daily Picks\n\n\u003e The following pairs are selected from the multi-timeframe trend scanner (Gate.io futures) and are for technical-analysis study only — not investment advice.\n\u003e Data updated: 2026-10-31 08:47:06\n\n### Long\n\n| Pair | Signal | Price | Take Profit | Stop Loss | R/R |\n|------|--------|-------|-------------|-----------|-----|\n| Q | 短平快做多 | $0.0434 | $0.0447 | $0.0423 | 1:1.2 |\n\n_1 picks selected. Scanner runs every 15 minutes.","positive_reactions_count":0,"cover_image":null,"tag_list":["partialtakeprofit","positionsizing","scalingout","trademanagement"],"canonical_url":"https://www.hkcode.dpdns.org/en/posts/33.html","reading_time_minutes":7,"user":{"name":"mattleeee","username":"mattleeee","twitter_username":null,"github_username":"mattleeee","user_id":4160029,"website_url":null,"profile_image":"https://media2.dev.to/dynamic/image/width=640,height=640,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png","profile_image_90":"https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png"}},{"type_of":"article","id":4944040,"title":"Circuit Breakers for Strategies: Pausing After Three Consecutive Losses","description":"Why Consecutive Losses Need Their Own Guard Drawdown limits answer \"how much have I lost?\"...","published":true,"published_at":"2026-10-30T01:02:05.567Z","slug":"circuit-breakers-for-strategies-pausing-after-three-consecutive-losses-5355","path":"/mattleeee/circuit-breakers-for-strategies-pausing-after-three-consecutive-losses-5355","url":"https://dev.to/mattleeee/circuit-breakers-for-strategies-pausing-after-three-consecutive-losses-5355","comments_count":0,"public_reactions_count":0,"page_views_count":20,"published_timestamp":"2026-10-30T01:02:05Z","body_markdown":"## Why Consecutive Losses Need Their Own Guard\n\nDrawdown limits answer \"how much have I lost?\" Consecutive-loss breakers answer a different question: \"is the process still behaving the way I validated it?\" A strategy can be down 4% over a month through a slow bleed of small wins and losses — normal. Or it can lose five trades in a row in two days — also possible under a valid edge, but far less likely, and much more informative. The breaker treats the streak as a signal about regime change, broken data feeds, or an implementation bug, not as a verdict on your equity curve.\n\nThe rule is deliberately blunt: three consecutive losing trades trips the strategy into cooldown. No auto-restart. A human reviews the trade history, checks whether the edge is still there, and re-arms the breaker explicitly.\n\n## The State Machine\n\nFour states, three transitions, one manual gate:\n\n

plaintext\nACTIVE ──(3 losses)──\u003e TRIPPED ──(notify + persist)──\u003e COOLDOWN\n ^ │\n └──────────(manual review / resume)────────────────────┘\n

\n\n- ACTIVE: strategy trades normally, counter increments on each closed loss and resets on each closed win.\n- TRIPPED: transient state entered the moment the third loss closes. Its only job is to fire the notification and persist the trip record. Execution must already be blocked here.\n- COOLDOWN: strategy is paused. No orders, no sizing, no \"just one small trade to test.\"\n- Resume: only via an explicit call after review. Never on a timer, never on a new day, never on \"market looks better now.\"\n\nThe important design decision is that TRIPPED blocks execution before the notification is sent. If the notification service is down, the strategy still doesn't trade. Failure to notify must never mean failure to stop.\n\n## Where the Counter Lives\n\nAn in-memory counter is worthless. Strategies get restarted by deploys, crashes, and Windows Task Scheduler reboots. If the counter resets on restart, a strategy that lost two trades before a restart gets a free pass on the third. Persist the counter and the state together, atomically, on every closed trade.\n\nSQLite is enough for a single-strategy setup and survives process death:\n\n

```python\nimport sqlite3\nfrom contextlib import contextmanager\nfrom dataclasses import dataclass\nfrom datetime import datetime, timezone\nfrom enum import Enum\n\nclass State(str, Enum):\n ACTIVE = \"active\"\n TRIPPED = \"tripped\"\n COOLDOWN = \"cooldown\"\n\n@dataclass\nclass BreakerRecord:\n strategy_id: str\n state: State\n consecutive_losses: int\n updated_at: str\n\nclass BreakerStore:\n def __init
(self, path: str = \"breaker.db\"):\n self.path = path\n with self._conn() as c:\n c.execute(\"\"\"\n CREATE TABLE IF NOT EXISTS breaker (\n strategy_id TEXT PRIMARY KEY,\n state TEXT NOT NULL,\n consecutive_losses INTEGER NOT NULL DEFAULT 0,\n updated_at TEXT NOT NULL\n )\n \"\"\")\n\n @contextmanager\n def _conn(self):\n conn = sqlite3.connect(self.path, isolation_level=None)\n try:\n conn.execute(\"PRAGMA journal_mode=WAL\")\n yield conn\n finally:\n conn.close()\n\n def load(self, strategy_id: str) -\u003e BreakerRecord:\n with self._conn() as c:\n row = c.execute(\n \"SELECT state, consecutive_losses, updated_at FROM breaker WHERE strategy_id=?\",\n (strategy_id,),\n ).fetchone()\n if row is None:\n return BreakerRecord(strategy_id, State.ACTIVE, 0, _now())\n return BreakerRecord(strategy_id, State(row[0]), row[1], row[2])\n\n def save(self, rec: BreakerRecord) -\u003e None:\n with self._conn() as c:\n c.execute(\"\"\"\n INSERT INTO breaker (strategy_id, state, consecutive_losses, updated_at)\n VALUES (?, ?, ?, ?)\n ON CONFLICT(strategy_id) DO UPDATE SET\n state=excluded.state,\n consecutive_losses=excluded.consecutive_losses,\n updated_at=excluded.updated_at\n \"\"\", (rec.strategy_id, rec.state.value, rec.consecutive_losses, rec.updated_at))\n\ndef _now() -\u003e str:\n return datetime.now(timezone.utc).isoformat()\n

\n\nTwo things matter here beyond persistence. First, `isolation_level=None` with WAL gives you autocommit plus concurrent readers, which is fine for this write pattern. Second, the state and the counter are one row — they can never disagree after a crash.\n\n## The Breaker Logic\n\nThe transition function is small and should stay that way. Every closed trade goes through it; nothing else mutates the counter.\n\n

python\nfrom dataclasses import replace\n\nMAX_CONSECUTIVE_LOSSES = 3\n\nclass LossBreaker:\n def __init
(self, store: BreakerStore, strategy_id: str,\n threshold: int = MAX_CONSECUTIVE_LOSSES):\n self.store = store\n self.strategy_id = strategy_id\n self.threshold = threshold\n\n def can_trade(self) -\u003e bool:\n return self.store.load(self.strategy_id).state is State.ACTIVE\n\n def on_trade_closed(self, pnl: float, trade_meta: dict) -\u003e BreakerRecord:\n rec = self.store.load(self.strategy_id)\n\n if rec.state is not State.ACTIVE:\n # Breaker is latched; closed PnL from any straggler orders\n # must not move the counter.\n return rec\n\n losses = rec.consecutive_losses + 1 if pnl \u003c 0 else 0\n new_state = State.TRIPPED if losses \u003e= self.threshold else State.ACTIVE\n rec = replace(rec, state=new_state, consecutive_losses=losses,\n updated_at=_now())\n self.store.save(rec)\n\n if new_state is State.TRIPPED:\n # Execution is already blocked by the persisted state.\n self._handle_trip(rec, trade_meta)\n return rec\n\n def _handle_trip(self, rec: BreakerRecord, trade_meta: dict) -\u003e None:\n history = self._recent_trades()\n # Move to COOLDOWN before notifying so a crash mid-notification\n # still leaves the strategy paused.\n self.store.save(replace(rec, state=State.COOLDOWN, updated_at=_now()))\n notify_trip(self.strategy_id, history, trade_meta)\n\n def _recent_trades(self) -\u003e list[dict]:\n # Pull the last N closed trades from your trade log.\n return load_recent_closed_trades(self.strategy_id, limit=10)\n\n def resume(self, reviewer: str, note: str) -\u003e BreakerRecord:\n rec = self.store.load(self.strategy_id)\n if rec.state is not State.COOLDOWN:\n raise RuntimeError(f\"cannot resume from state {rec.state}\")\n log_review(self.strategy_id, reviewer, note, rec.consecutive_losses)\n rec = replace(rec, state=State.ACTIVE, consecutive_losses=0,\n updated_at=_now())\n self.store.save(rec)\n return rec\n

\n\nNote the ordering in `_handle_trip`: persist `COOLDOWN` first, then notify. If `notify_trip` throws, the strategy is already stopped. The notification is a side effect, not part of the safety mechanism.\n\nAlso note that `on_trade_closed` ignores PnL while the breaker is latched. This handles the real-world case where a stop order fills after the trip — you don't want that fill to silently reset a loss counter or, worse, re-trip an already-tripped breaker and re-fire notifications.\n\n## Wiring It Into the Execution Loop\n\nThe check has to sit in front of order submission, not inside the strategy logic. Strategy code is where you're most tempted to add exceptions.\n\n

python\ndef run_bar(breaker: LossBreaker, strategy, market_data) -\u003e None:\n if not breaker.can_trade():\n return # paused: no signals, no orders, no sizing\n\n signal = strategy.evaluate(market_data)\n if signal is None:\n return\n\n fill = execute_order(signal)\n if fill is not None and fill.closed:\n breaker.on_trade_closed(fill.pnl, fill.meta)\n

\n\nIf your strategy emits orders from multiple places, wrap `execute_order` itself with the `can_trade()` guard as a second line of defense. Belt and suspenders is cheap here.\n\n## Notifications That Are Actually Useful\n\nA push that says \"strategy tripped\" wastes the meeting. The notification should carry the evidence: the losing trades, their entry/exit, size, PnL, and the timestamps. That's what makes the review a review instead of a shrug.\n\n

python\nimport json\nimport requests\n\ndef notify_trip(strategy_id: str, history: list[dict], trigger: dict) -\u003e None:\n lines = [f\"Circuit breaker tripped: {strategy_id}\", \"\"]\n for t in history:\n lines.append(\n f\"{t['closed_at']} {t['side']:\u003c4} size={t['size']:\u003c8} \"\n f\"pnl={t['pnl']:+.2f} {t.get('exit_reason', '')}\"\n )\n lines.append(\"\")\n lines.append(f\"Triggering trade: {json.dumps(trigger, default=str)}\")\n body = \"\n\".join(lines)\n\n requests.post(\n \"https://ntfy.sh/your-private-topic\",\n data=body.encode(\"utf-8\"),\n headers={\"Title\": f\"BREAKER: {strategy_id}\", \"Priority\": \"high\"},\n timeout=10,\n )\n``

\n\nKeep the payload small and text-based. It should be readable on a phone lock screen. If you use a different channel — email, Telegram, a local webhook — the shape stays the same: strategy id, the streak, the trades.\n\n## Consecutive Losses vs. Drawdown Limits\n\nThese are not substitutes, and running only one is a common gap.\n\n| | Drawdown limit | Consecutive-loss breaker |\n|---|---|---|\n| Measures | Cumulative capital loss | Process deviation |\n| Typical trigger | Slow bleed over weeks | Fast streak over days |\n| Reset | New equity high | A single win |\n| Failure mode it catches | Sizing too large for the edge | Edge decay, feed corruption, bug |\n| Response | Reduce size / stop | Pause and review |\n\nA drawdown limit won't fire on three losses if the prior month was strong. A consecutive-loss breaker will. Conversely, a breaker won't catch a strategy that loses 55% of trades in a steady grind — you need the drawdown limit for that. Run both. They answer different questions and they fail in different directions.\n\nOne more distinction worth stating: the breaker is per strategy, not per account. If you run five strategies and one trips, the other four should keep running. Coupling them turns a single-strategy problem into a portfolio-wide outage.\n\n## The Psychology Part\n\nThe breaker is not punishment. It's a forced meeting with your own statistics.\n\nThree losses in a row is not rare. For a strategy with a 45% win rate, the probability of any given three-trade window being all losses is about 16.6%. Over a few hundred trades you'll see it constantly. That's precisely why the rule is mechanical: if you decide case-by-case whether a streak is \"just variance,\" you will decide it's variance every time, because that's the answer that lets you keep trading.\n\nThe manual resume step is the actual feature. It forces you to write down what you saw — the trades, the market context, whether the fills looked normal — before the strategy goes live again. Sometimes the review takes two minutes and you're back. Sometimes it takes a week and the strategy gets retired. Both are correct outcomes. What's not correct is a timer that restarts everything at midnight and calls it risk management.\n\nTwo practical habits make reviews fast:\n\n1. **Log the review.** The
log_review call in resume() writes who resumed, when, and why. After a few months you have a record of which strategies trip often and whether those trips preceded real problems. That's data about your breaker, not just your strategy.\n2. **Tune the threshold per strategy, not globally.** Three is a reasonable default for strategies trading a few times a day. For a high-frequency strategy, three losses is noise; use a larger threshold or a percentage-based variant. For a low-frequency strategy, three losses might span months — there, a drawdown limit does more work.\n\n## Adapting It\n\nThe same structure generalizes cheaply:\n\n- **Percentage-based streak**: trip when cumulative loss over the current streak exceeds *x*% of equity, instead of a fixed count.\n- **Time-bounded streak**: only count losses within a rolling window, so a loss in January and a loss in June aren't \"consecutive.\"\n- **Per-instrument breakers**: key the store on strategy_id + symbol` if your strategy trades several instruments with independent behavior.\n\nIn all cases, keep the same invariants: persist state and counter together, block execution before notifying, and require an explicit, logged resume.\n\nMore notes like this ship every week on this site.\n\n---\n\n## Daily Picks\n\n\u003e The following pairs are selected from the multi-timeframe trend scanner (Gate.io futures) and are for technical-analysis study only — not investment advice.\n\u003e Data updated: 2026-10-30 08:46:53\n\n### Long\n\n| Pair | Signal | Price | Take Profit | Stop Loss | R/R |\n|------|--------|-------|-------------|-----------|-----|\n| TRX | 高质量标的做多 | $0.3521 | $0.3768 | $0.3398 | 1:2.0 |\n\n### Short\n\n| Pair | Signal | Price | Take Profit | Stop Loss | R/R |\n|------|--------|-------|-------------|-----------|-----|\n| SNXX | 庄家做空 | $8.496 | $8.0967 | $8.7934 | 1:1.3 |\n\n_2 picks selected. Scanner runs every 15 minutes.
","positive_reactions_count":0,"cover_image":null,"tag_list":["circuitbreaker","consecutivelosses","cooldown","riskcontrol"],"canonical_url":"https://www.hkcode.dpdns.org/en/posts/32.html","reading_time_minutes":7,"user":{"name":"mattleeee","username":"mattleeee","twitter_username":null,"github_username":"mattleeee","user_id":4160029,"website_url":null,"profile_image":"https://media2.dev.to/dynamic/image/width=640,height=640,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png","profile_image_90":"https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png"}},{"type_of":"article","id":4937527,"title":"The Resonance Scanner: Seven Filters and the One That Mattered","description":"The Resonance Scanner: Seven Filters and the One That Mattered A scanner is a conjunction....","published":true,"published_at":"2026-10-29T01:02:11.309Z","slug":"the-resonance-scanner-seven-filters-and-the-one-that-mattered-223e","path":"/mattleeee/the-resonance-scanner-seven-filters-and-the-one-that-mattered-223e","url":"https://dev.to/mattleeee/the-resonance-scanner-seven-filters-and-the-one-that-mattered-223e","comments_count":0,"public_reactions_count":0,"page_views_count":0,"published_timestamp":"2026-10-29T01:02:11Z","body_markdown":"## The Resonance Scanner: Seven Filters and the One That Mattered\n\nA scanner is a conjunction. Every filter must pass, so the output rate is the product of the individual pass rates. This is obvious arithmetic and almost nobody instruments for it. Instead, we tune filters one at a time, looking at each in isolation, and feel good about every one of them. \"The trend filter is correct. The volume filter is correct. The momentum filter is correct.\" Seven correct filters, zero positions.\n\nThat was the situation with our resonance scanner: a seven-layer entry gate that produced watchlist entries for weeks and never opened a position. The forensics were not subtle once we looked. The binding constraint was pullback confirmation, the last layer, and the reason it bound was structural, not statistical: the moves we were trying to catch do not, by their nature, offer a clean three-candle pullback.\n\n### Anatomy of the Gate\n\nThe pipeline looked like this, in order:\n\n1. Trend — higher-timeframe EMA alignment.\n2. Volume — relative volume above a rolling baseline.\n3. Momentum — rate-of-change percentile.\n4. Volatility — ATR band, to skip dead and berserk regimes.\n5. Session — allowed trading hours.\n6. Blacklist — symbol exclusion list.\n7. Pullback confirmation — a defined retracement over N candles, then a resumption candle.\n\nEach filter is a pure function of a dataframe slice, which makes the whole thing easy to test and easy to instrument. The entry point was a single evaluate() that short-circuited on the first failure. That short-circuit was efficient and it was also why we were blind: a candidate rejected at layer 2 never touched layers 3 through 7, so we had no idea how many candidates each layer would have killed independently.\n\n## Instrumenting the Filters\n\nThe fix for blindness is cheap. Keep the short-circuit for live use, but add a shadow pass that evaluates every filter regardless of earlier failures and records the result. Two things matter in the record: the per-filter pass boolean, and the order of the first failure. The second is what tells you which filter is binding.\n\n

python\nfrom dataclasses import dataclass, field\n\n@dataclass\nclass FilterTrace:\n symbol: str\n ts: int\n results: dict = field(default_factory=dict) # name -\u003e bool\n\n @property\n def first_failure(self):\n for name, passed in self.results.items(): # insertion order = pipeline order\n if not passed:\n return name\n return None\n\n\ndef shadow_evaluate(symbol, ts, ctx, filters):\n \"\"\"Evaluate all filters without short-circuiting. Order preserved.\"\"\"\n trace = FilterTrace(symbol=symbol, ts=ts)\n for name, fn in filters: # list of (name, callable), in pipeline order\n try:\n trace.results[name] = bool(fn(ctx))\n except Exception:\n trace.results[name] = False # fail closed, but record it\n return trace\n

\n\nNote the except clause. A filter that throws is a filter that rejected, and silently swallowing the exception is how you end up with a layer that never passes and nobody notices. Fail closed, but count it separately so exceptions do not masquerade as legitimate rejections.\n\nLive evaluation stays fast:\n\n

python\ndef live_evaluate(ctx, filters):\n for name, fn in filters:\n if not fn(ctx):\n return False, name\n return True, None\n

\n\nBoth paths share the same filter list, so they cannot drift apart.\n\n### Aggregating the Traces\n\nWith a few weeks of traces stored, the analysis is a groupby. For every filter we want the independent pass rate (how often it would pass if it were the only gate), and the conditional pass rate given that all earlier filters passed. The gap between those two numbers is where the interesting failures live.\n\n

python\nimport pandas as pd\n\ndef filter_report(traces: list[FilterTrace]) -\u003e pd.DataFrame:\n rows = []\n for t in traces:\n rows.append({\"symbol\": t.symbol, \"ts\": t.ts,\n \"first_failure\": t.first_failure, **t.results})\n df = pd.DataFrame(rows)\n\n order = list(traces[0].results.keys())\n out = []\n for i, name in enumerate(order):\n independent = df[name].mean()\n if i == 0:\n conditional = independent\n reached = 1.0\n else:\n prior = order[:i]\n reached_mask = df[prior].all(axis=1)\n reached = reached_mask.mean()\n conditional = df.loc[reached_mask, name].mean() if reached_mask.any() else float(\"nan\")\n out.append({\n \"filter\": name,\n \"independent_pass\": round(independent, 4),\n \"reached_rate\": round(reached, 4),\n \"conditional_pass\": round(conditional, 4),\n })\n return pd.DataFrame(out)\n

\n\nreached_rate is the fraction of candidates that survived everything before this layer. conditional_pass is the pass rate among those that got there. A filter with a high independent pass rate and a low conditional pass rate is doing real work at the margin. A filter with a low independent pass rate is a candidate for removal regardless of where it sits.\n\nThen the binding constraint is simply the mode of first_failure:\n\n

python\nbinding = df[\"first_failure\"].value_counts(dropna=False)\nprint(binding.head())\n

\n\nOur output looked roughly like this (numbers rounded, structure exact):\n\n| filter | independent_pass | reached_rate | conditional_pass |\n|---|---|---|---|\n| trend | 0.31 | 1.00 | 0.31 |\n| volume | 0.44 | 0.31 | 0.52 |\n| momentum | 0.38 | 0.16 | 0.61 |\n| volatility | 0.72 | 0.10 | 0.79 |\n| session | 0.83 | 0.08 | 0.88 |\n| blacklist | 0.97 | 0.07 | 0.98 |\n| pullback | 0.11 | 0.07 | 0.11 |\n\nRead the last row carefully. Seven percent of candidates reached the pullback layer, and only eleven percent of those passed it. The first six filters were, individually and collectively, behaving as designed: they cut a large universe down to a small, sensible set of live movers. The seventh filter then threw away 89% of that set. The mode of first_failure confirmed it: pullback was the last rejection for the overwhelming majority of candidates that got that far.\n\n## The Structural Problem\n\nThe naive conclusion is \"loosen the pullback filter.\" That would have been wrong, and it took a while to articulate why.\n\nThe scanner's thesis was resonance: trend, volume, and momentum aligned on a symbol that was being actively accumulated. In practice, the moves that satisfied the first six filters were fast, high-volume expansions, often on altcoins during a coordinated push. These moves have a characteristic shape: a steep impulse, a shallow consolidation or a single red candle, then continuation or exhaustion. What they almost never have is a textbook three-candle retracement followed by a clean resumption candle. The pullback filter was written for a different market behavior: orderly trends that breathe. By the time a clean three-candle pullback prints on a fast expansion, the expansion is over. The filter was not rejecting noise. It was rejecting the exact events the scanner existed to find.\n\nThis is the general failure mode of stacked filters: each one is validated against the full population, but the conjunction selects a subpopulation with different conditional statistics. A pullback filter that works beautifully on a broad universe of trending names can have near-zero pass rate on the narrow slice that survives six upstream gates, because the upstream gates have already selected for the behavior the pullback filter excludes. Filters are not independent, and a pipeline is not a bag of good ideas.\n\n## The Design Tradeoff\n\nThe real decision was not \"keep or drop pullback.\" It was where to sit on the spectrum between few but clean and so strict it never trades.\n\nThe discipline argument for strictness is sound. Every filter you add is a degree of freedom, and every degree of freedom is an opportunity to fit the last regime. A seven-condition entry is nearly impossible to overfit by accident, because the search space of ways to be wrong is enormous and the sample of trades is tiny. If a strict system trades rarely, at least the trades it takes are well-supported.\n\nThe counterargument is that a scanner which never trades is not conservative, it is broken. It provides no information, accumulates no live sample, and its filters are never validated against real outcomes. You cannot distinguish a disciplined system from a mis-specified one if the mis-specification happens to be the last gate in the chain. Strictness that produces zero fills is not a virtue; it is a measurement failure wearing a virtue's clothes.\n\nThe resolution we landed on was to treat the pullback layer as a regime-conditional filter rather than a universal one, and to replace the hard three-candle rule with a graded score so the layer can be strict in one regime and permissive in another without being deleted outright.\n\n

python\ndef pullback_score(df, lookback=3, min_retrace=0.25, max_retrace=0.75):\n \"\"\"\n Score 0..1 instead of boolean. Measures whether the recent action\n shows a controlled retracement rather than a vertical run.\n \"\"\"\n impulse = df[\"close\"].iloc[-lookback-1] - df[\"close\"].iloc[-lookback*2-1]\n if impulse \u003c= 0:\n return 0.0\n retrace = (df[\"close\"].iloc[-lookback-1] - df[\"close\"].iloc[-1]) / impulse\n if retrace \u003c min_retrace:\n return 0.0 # no pullback at all: vertical run\n if retrace \u003e max_retrace:\n return 0.0 # too deep: structure broken\n # peak score at mid-retrace, taper toward both edges\n mid = (min_retrace + max_retrace) / 2\n width = (max_retrace - min_retrace) / 2\n return max(0.0, 1.0 - abs(retrace - mid) / width)\n\n\ndef pullback_gate(ctx, threshold=0.5):\n score = pullback_score(ctx.df)\n # Fast-expansion regime: accept a shallower structure.\n if ctx.regime == \"expansion\":\n return score \u003e= threshold * 0.6\n return score \u003e= threshold\n

\n\nTwo changes matter here. First, the layer now returns a continuous score, so you can see how close near-misses were instead of only pass/fail. Second, the threshold is regime-dependent, which encodes the actual finding: fast expansions legitimately do not pull back like orderly trends do, and pretending otherwise is a modeling error, not a discipline.\n\nCrucially, we did not just lower the threshold globally until trades appeared. That is the opposite failure: tuning a filter until it stops binding, then calling the result validated. The regime split came from the trace data (pullback pass rates differed sharply by regime upstream), and the new threshold was chosen on out-of-sample traces, not on the same window that produced the diagnosis.\n\n### What to Keep\n\nThe instrumented pipeline is worth keeping permanently, not just for this investigation. The FilterTrace records cost almost nothing, and the filter_report table is the single most useful diagnostic a multi-condition scanner can have. Run it monthly. If the binding constraint moves, you have learned something about the market. If it never moves, you have learned something about your filters: one of them is doing all the work, and the other six are decoration.\n\nThe lesson generalizes past this scanner. A conjunction of filters is only as good as its weakest link, and you cannot find the weakest link by inspecting filters individually. You find it by counting first failures, computing conditional pass rates, and asking whether the surviving subpopulation is the one you actually meant to trade. The filter that rejects last is the filter that defines your system, whether or not you intended it to.\n\nMore notes like this ship every week on this site.\n\n---\n\n## Daily Picks\n\n\u003e The following pairs are selected from the multi-timeframe trend scanner (Gate.io futures) and are for technical-analysis study only — not investment advice.\n\u003e Data updated: 2026-10-29 08:46:59\n\n### Long\n\n| Pair | Signal | Price | Take Profit | Stop Loss | R/R |\n|------|--------|-------|-------------|-----------|-----|\n| BTC | 趋势低吸做多 | $81120.0 | $87204.0 | $80574.0602 | 1:11.1 |\n| SNXX | 高质量标的做多 | $10.016 | $10.3866 | $9.8157 | 1:1.9 |\n\n_2 picks selected. Scanner runs every 15 minutes.","positive_reactions_count":0,"cover_image":null,"tag_list":["scanner","signalfilters","multiconditionentry","designtradeoff"],"canonical_url":"https://www.hkcode.dpdns.org/en/posts/31.html","reading_time_minutes":7,"user":{"name":"mattleeee","username":"mattleeee","twitter_username":null,"github_username":"mattleeee","user_id":4160029,"website_url":null,"profile_image":"https://media2.dev.to/dynamic/image/width=640,height=640,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png","profile_image_90":"https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png"}},{"type_of":"article","id":4931325,"title":"A/B Testing Trading Rules: The 52 Rejected Trades That Changed My Mind","description":"The Question a Filter Should Answer A pullback filter sits in front of an entry signal and...","published":true,"published_at":"2026-10-28T01:01:52.676Z","slug":"ab-testing-trading-rules-the-52-rejected-trades-that-changed-my-mind-3kop","path":"/mattleeee/ab-testing-trading-rules-the-52-rejected-trades-that-changed-my-mind-3kop","url":"https://dev.to/mattleeee/ab-testing-trading-rules-the-52-rejected-trades-that-changed-my-mind-3kop","comments_count":0,"public_reactions_count":0,"page_views_count":40,"published_timestamp":"2026-10-28T01:01:52Z","body_markdown":"## The Question a Filter Should Answer\n\nA pullback filter sits in front of an entry signal and decides which setups are allowed through. The usual debate goes like this: the strict variant looks cleaner on the equity curve, the relaxed variant trades more, and someone argues the extra trades are \"noise.\" That argument is almost never tested. It is asserted from the shape of the curve.\n\nThe right question is narrower: what happens to the trades that exist only because of the relaxed rule? If that incremental set has positive expectancy, the strict filter is not cleaning up noise. It is deleting the strategy's core. In the case I worked through, the strict rule rejected 52 trades that the relaxed rule took. Those 52 trades had a 48.1% win rate, +0.131U expectancy per trade, and a profit factor of 1.50. The \"worse\" rule was rejecting the profitable part.\n\nThis post is about the method, not the specific rule: run both variants over the same window, diff the trade lists, and evaluate the rejected set on its own statistics.\n\n## Why the Headline Curve Can't Settle It\n\nTwo variants produce two equity curves. Comparing them tells you which variant made more money over one sample. It does not tell you whether the filter is doing what you think it is doing, because the curves aggregate everything: shared trades, rejected trades, and the timing of both.\n\nA strict filter can improve a curve for three different reasons:\n\n1. It removes genuinely bad trades (the intended effect).\n2. It removes trades that were fine but happened to lose in this sample (overfitting to noise).\n3. It changes position sizing or trade sequencing in a way that masks the removal of good trades.\n\nOnly the first is a reason to keep it. The diff isolates the trades that differ, and their statistics tell you which of the three you are looking at.\n\n### The diff is the unit of analysis\n\nThink of the two variants as generating two sets of trades over an identical window. The shared trades are irrelevant to the filter decision — both rules took them, so they contribute equally to both curves. The decision lives entirely in the symmetric difference:\n\n- Rejected-only: trades the strict rule skipped and the relaxed rule took.\n- Accepted-only: trades the relaxed rule skipped and the strict rule took (this happens when the rules are not strictly nested).\n\nIf the rejected-only set has positive expectancy with a reasonable sample size, the filter is destroying value. If it has strongly negative expectancy, the filter is earning its keep. If the set is tiny, you have no evidence either way and should not pretend otherwise.\n\n## Building the Two Trade Lists\n\nThe prerequisite is that both variants log trades in the same schema, over the same data, with the same execution assumptions. Any difference in slippage model or fill logic between runs will contaminate the diff.\n\n

python\nfrom dataclasses import dataclass\nfrom datetime import datetime\n\n@dataclass(frozen=True)\nclass Trade:\n symbol: str\n entry_time: datetime\n exit_time: datetime\n direction: str # \"long\" / \"short\"\n entry_price: float\n exit_price: float\n size: float # units\n pnl: float # realized, in account currency\n rule_variant: str # \"strict\" / \"relaxed\"\n

\n\nA trade key needs to identify the same opportunity across runs. Entry timestamp plus symbol plus direction is usually enough; if a strategy can re-enter the same symbol on the same bar, add an entry-sequence index.\n\n

python\ndef trade_key(t: Trade) -\u003e tuple:\n return (t.symbol, t.entry_time, t.direction)\n

\n\nRun both variants over the same window and the same instrument universe:\n\n

python\nstrict_trades = run_backtest(strategy, data, params=STRICT_PARAMS)\nrelaxed_trades = run_backtest(strategy, data, params=RELAXED_PARAMS)\n\nstrict_by_key = {trade_key(t): t for t in strict_trades}\nrelaxed_by_key = {trade_key(t): t for t in relaxed_trades}\n\nrejected_only = [t for k, t in relaxed_by_key.items() if k not in strict_by_key]\naccepted_only = [t for k, t in strict_by_key.items() if k not in relaxed_by_key]\nshared = [t for k, t in strict_by_key.items() if k in relaxed_by_key]\n

\n\nSanity check before analyzing anything: len(shared) + len(rejected_only) should equal the relaxed run's trade count, and the PnL of the shared set should be identical in both runs. If it is not, the two runs are not comparable and the diff means nothing.\n\n## Statistics for the Incremental Set\n\nThe rejected-only set is just a list of trades. Evaluate it exactly as you would a standalone strategy, but report the full picture, not one number.\n\n

python\nimport statistics as st\n\ndef summarize(trades: list[Trade]) -\u003e dict:\n n = len(trades)\n if n == 0:\n return {\"n\": 0}\n\n pnls = [t.pnl for t in trades]\n wins = [p for p in pnls if p \u003e 0]\n losses = [p for p in pnls if p \u003c 0]\n\n gross_win = sum(wins)\n gross_loss = -sum(losses)\n\n return {\n \"n\": n,\n \"win_rate\": len(wins) / n,\n \"avg_win\": st.mean(wins) if wins else 0.0,\n \"avg_loss\": st.mean(losses) if losses else 0.0,\n \"expectancy\": st.mean(pnls),\n \"profit_factor\": gross_win / gross_loss if gross_loss \u003e 0 else float(\"inf\"),\n \"total_pnl\": sum(pnls),\n }\n

\n\nFor the 52 rejected trades this returns something like:\n\n

text\nn=52 win_rate=0.481 avg_win=+0.71U avg_loss=-0.41U\nexpectancy=+0.131U profit_factor=1.50 total_pnl=+6.81U\n

\n\nRead it as a whole. A 48.1% win rate sounds unremarkable until you pair it with the payoff ratio: average win is roughly 1.7x average loss, so sub-50% accuracy still yields positive expectancy. Profit factor 1.50 over 52 trades is not spectacular, but it is clearly the wrong side of zero for a filter that is supposed to be removing bad trades. A filter that rejects a set with PF 1.50 is rejecting trades you would have wanted.\n\n### Interpret the shape, not just the sign\n\nThree patterns are worth distinguishing when you look at the rejected-only set:\n\n- Negative expectancy with a normal win rate: the filter is removing losers. Keep it.\n- Near-zero expectancy with a large sample: the filter is removing noise. Keep it only if it materially reduces drawdown or costs; otherwise it is not earning its complexity.\n- Positive expectancy: the filter is removing profit. The strict variant needs a different justification, or it should go.\n\nThe 52-trade case is the third pattern, and the win rate alone would have hidden it. This is why the diff has to be evaluated with expectancy and profit factor, not just hit rate.\n\n## Guarding Against One Lucky Trade\n\nThe most common way a diff analysis lies is concentration. Fifty-two trades with +0.131U expectancy can be twelve good trades and forty flat ones, or one trade that made the entire set. Before trusting the result, check:\n\n### Per-trade contribution\n\n

python\ndef concentration_report(trades: list[Trade], top_k: int = 5) -\u003e dict:\n pnls = sorted((t.pnl for t in trades), reverse=True)\n total = sum(pnls)\n top = sum(pnls[:top_k])\n\n return {\n \"total_pnl\": total,\n \"top_k_pnl\": top,\n \"top_k_share\": top / total if total != 0 else float(\"inf\"),\n \"expectancy_ex_top_k\": (total - top) / (len(pnls) - top_k) if len(pnls) \u003e top_k else None,\n }\n

\n\nIf the top five trades account for most of the set's PnL and expectancy excluding them collapses toward zero, the incremental set is not evidence of a better rule. It is evidence of a handful of outcomes.\n\n### Per-symbol breakdown\n\nThe diff can also be dominated by one instrument. Group the rejected-only trades by symbol and compare each group's count and expectancy against its share of the full sample.\n\n

python\nfrom collections import defaultdict\n\ndef by_symbol(trades: list[Trade]) -\u003e dict[str, dict]:\n groups = defaultdict(list)\n for t in trades:\n groups[t.symbol].append(t)\n return {sym: summarize(ts) for sym, ts in groups.items()}\n

\n\nIf one symbol supplies most of the rejected trades and all of the positive expectancy, the honest conclusion is \"the relaxed rule helps on this symbol in this window,\" not \"the relaxed rule is better.\" A filter decision made on a single-symbol diff is a single-symbol decision.\n\n### Sample size and stability\n\nFifty-two trades is enough to see a signal but not enough to be confident about its magnitude. Split the rejected-only set in half by time and check whether expectancy holds in both halves. If the first half carries everything and the second half is flat or negative, the incremental set is not a stable property of the rule.\n\n

python\ndef split_half(trades: list[Trade]) -\u003e tuple[list[Trade], list[Trade]]:\n ordered = sorted(trades, key=lambda t: t.entry_time)\n mid = len(ordered) // 2\n return ordered[:mid], ordered[mid:]\n

\n\n## What the Method Actually Decides\n\nThe diff analysis does not tell you which rule is \"correct.\" It tells you what each rule is doing. In this case, the strict pullback filter was not removing noise — it was removing a set of trades with positive expectancy, a 1.50 profit factor, and a payoff profile that made a sub-50% win rate profitable. The relaxed rule was not sloppier; the strict rule was over-filtered.\n\nThe general procedure generalizes beyond pullback filters. Any time you compare two entry conditions, two exit rules, or two position-sizing schemes, the shared trades cancel and the decision lives in the diff. Evaluate the incremental set with count, win rate, average win and loss, expectancy, and profit factor, then check whether that set is concentrated in a few trades or a single symbol. If it is, you have a hypothesis, not a result.\n\nThe headline curve is a summary of everything at once. The diff is the only place the filter's actual effect is visible.\n\nMore notes like this ship every week on this site.","positive_reactions_count":0,"cover_image":null,"tag_list":["abtest","incrementaltrades","profitfactor","winrate"],"canonical_url":"https://www.hkcode.dpdns.org/en/posts/30.html","reading_time_minutes":6,"user":{"name":"mattleeee","username":"mattleeee","twitter_username":null,"github_username":"mattleeee","user_id":4160029,"website_url":null,"profile_image":"https://media2.dev.to/dynamic/image/width=640,height=640,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png","profile_image_90":"https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png"}},{"type_of":"article","id":4924883,"title":"Backtesting the Pullback Filter: When a Safety Check Kills Profit","description":"The Setup: One Strategy, Two Filters The base strategy is a long-only momentum entry on...","published":true,"published_at":"2026-10-27T01:02:36.311Z","slug":"backtesting-the-pullback-filter-when-a-safety-check-kills-profit-4439","path":"/mattleeee/backtesting-the-pullback-filter-when-a-safety-check-kills-profit-4439","url":"https://dev.to/mattleeee/backtesting-the-pullback-filter-when-a-safety-check-kills-profit-4439","comments_count":0,"public_reactions_count":1,"page_views_count":21,"published_timestamp":"2026-10-27T01:02:36Z","body_markdown":"## The Setup: One Strategy, Two Filters\n\nThe base strategy is a long-only momentum entry on 15-minute candles. Signal logic, position sizing, and exit rules are fixed. The only thing that changes between variants is the pullback confirmation that must be true before an entry fires.\n\nThe strict filter requires all three of the following on the entry bar:\n\n1. At least one down candle among the last three 15-minute candles.\n2. Current price below the high of the last five candles.\n3. The latest candle is not a large bullish candle (body above a threshold, e.g. 0.8 × ATR).\n\nThe relaxed filter drops the three-candle window and the big-bullish veto, keeping only: at least one down candle within the last five candles. It still demands a pullback, just a looser one.\n\nBoth variants run on the same 90-day window, 24 symbols, same fee model, same slippage assumption. That is the whole point of the experiment: a single boolean changes, everything else is held constant.\n\n## Why the Comparison Is Easy to Get Wrong\n\nThe tempting move is to compare the two versions by their win rates or average PnL per trade. That comparison is contaminated. The strict filter does not just change how trades are taken — it changes which trades exist. The strict set is a subset of the relaxed set, so you are comparing two different populations of trades, not the same trades under two treatments.\n\nTo say anything honest about the filter, you need to look at the trades the strict filter rejects. Those rejected entries are where the answer lives. If the rejected trades have positive expectancy on their own, the filter is destroying edge. If they are negative, the filter is earning its keep.\n\nSo the experiment has three outputs, not two:\n\n- Trades taken by both variants (the intersection).\n- Trades taken only by the relaxed variant (rejected by strict).\n- Expectancy of each group.\n\nThe base strategy under the relaxed filter is the control population. The strict filter is a treatment applied to the signal stream. Treat it like an A/B test where the \"B\" arm is a subset of \"A\", and you want to know whether the excluded slice is better or worse than average.\n\n## Counting the Incremental Trades\n\nThe cleanest way to run this is to evaluate the signal stream once, tag every candidate entry with the filter booleans, and then slice. Here is a minimal harness that does exactly that.\n\n

python\nfrom dataclasses import dataclass\nfrom typing import Callable\n\n@dataclass\nclass Trade:\n symbol: str\n ts: int\n entry: float\n exit: float\n pnl: float # net of fees and slippage, in quote units\n\ndef strict_pullback(candles, i: int, atr: float) -\u003e bool:\n # candles: list of (ts, o, h, l, c) ascending, i = entry bar index\n last3 = candles[i-2:i+1]\n has_down = any(c[4] \u003c c[1] for c in last3)\n five_high = max(c[2] for c in candles[i-4:i+1])\n body = candles[i][4] - candles[i][1]\n not_big_bull = body \u003c 0.8 * atr\n return has_down and candles[i][4] \u003c five_high and not_big_bull\n\ndef relaxed_pullback(candles, i: int, atr: float) -\u003e bool:\n last5 = candles[i-4:i+1]\n return any(c[4] \u003c c[1] for c in last5)\n

\n\nThe backtest engine stays untouched. You generate candidate entries from the momentum signal, then record both filter verdicts alongside the realized PnL. Note that the entry price and exit price are identical for a given candidate regardless of which filter accepted it — the filter does not move the entry bar, it only decides whether the bar is tradeable at all. That assumption matters: if your strict filter also shifts entry timing (e.g. waiting for confirmation on the next bar), you have changed two variables and the comparison is dead.\n\n

python\ndef run_backtest(candles_by_symbol, signal_fn, exit_fn):\n rows = []\n for sym, candles in candles_by_symbol.items():\n atr_series = compute_atr(candles, 14)\n for i in range(20, len(candles) - 1):\n if not signal_fn(candles, i):\n continue\n atr = atr_series[i]\n entry = candles[i][4]\n exit_price = exit_fn(candles, i)\n pnl = (exit_price - entry) - fee_and_slip(entry, exit_price)\n rows.append({\n \"symbol\": sym,\n \"ts\": candles[i][0],\n \"pnl\": pnl,\n \"strict\": strict_pullback(candles, i, atr),\n \"relaxed\": relaxed_pullback(candles, i, atr),\n })\n return rows\n

\n\nNow the analysis is just a groupby. No second backtest run, no risk of drift between the two code paths.\n\n

python\ndef slice_expectancy(rows):\n both = [r[\"pnl\"] for r in rows if r[\"strict\"] and r[\"relaxed\"]]\n strict_only = [r[\"pnl\"] for r in rows if r[\"strict\"] and not r[\"relaxed\"]]\n relaxed_only = [r[\"pnl\"] for r in rows if r[\"relaxed\"] and not r[\"strict\"]]\n # note: strict implies relaxed by construction only if the\n # relaxed rule is a strict superset — verify this before trusting it\n return {\n \"both\": (len(both), sum(both) / len(both) if both else 0),\n \"strict_rejected\": (len(relaxed_only),\n sum(relaxed_only) / len(relaxed_only) if relaxed_only else 0),\n \"strict_only\": (len(strict_only),\n sum(strict_only) / len(strict_only) if strict_only else 0),\n }\n

\n\nOne check before you trust any of this: confirm the strict rule is actually a subset of the relaxed rule. If your relaxed rule is \"any down candle in the last five\" and your strict rule also requires \"price below the five-candle high,\" then strict does not imply relaxed — a bar can satisfy strict's down-candle condition and high condition while failing nothing in relaxed, but the reverse can also happen if the down candle is outside the five-bar window in strict's three-bar check. Run the set-difference both ways and print the counts. If strict_only is non-empty, your two rules are not nested, and you should compare them as two separate populations rather than a control and a treatment.\n\n## The Numbers\n\nOn the 90-day, 24-symbol run:\n\n| Variant | Trades | Expectancy (U/trade) |\n|---|---|---|\n| Strict pullback | ~1,900 | -0.013 |\n| Relaxed pullback | ~2,600 | +0.011 |\n\nThe strict filter rejects roughly 700 candidate entries. Those rejected trades carry positive expectancy — enough to flip the whole strategy from negative to positive. The filter did not remove noise; it removed the profitable tail.\n\nThe win rate on the strict set was actually slightly higher. That is the trap. A filter that raises win rate while lowering expectancy is selecting for small, frequent wins and occasional large losses — exactly what a late, tired move produces.\n\n## Why \"Safer\" Is Worse Here\n\nThe strict filter's three conditions all bias toward the same failure mode: entering after the move has already extended and then rolled over.\n\n- Requiring a down candle in the last three bars means you enter on the first sign of weakness. In a momentum regime, that is often the start of a real reversal, not a healthy pullback.\n- Requiring price below the five-candle high means you are buying into a local dip — but only after the high was already set. The move is mature.\n- Vetoing a big bullish candle removes exactly the bars where momentum is re-accelerating. Those are the entries with the best follow-through.\n\nThe relaxed filter keeps the pullback requirement but gives the move room to breathe. It accepts entries where the pullback happened four or five bars ago and price has since resumed. Those entries are earlier in the next leg, not later in the current one.\n\nPut differently: the strict filter is a proxy for \"the setup looks clean right now.\" But clean-looking setups are, by construction, the ones where the crowd has already noticed. The relaxed filter accepts messier setups that the strategy's exit logic can still harvest. The filter was supposed to reduce drawdown. It reduced trade count and took the edge with it.\n\n## Practical Rules for Filter A/B Tests\n\nA few habits that keep this kind of test honest:\n\n*Keep the exit fixed.* If you tune the exit for each filter variant, you are no longer measuring the filter. Any difference in expectancy is confounded by the exit change.\n\n*Always report the rejected slice.* A filter's value is the expectancy of what it excludes. If you do not compute that number, you do not know what the filter does.\n\n*Check nesting before slicing.* Confirm which candidate population each rule accepts, and whether one is a subset of the other. Non-nested rules need separate baselines.\n\n*Watch win rate and expectancy diverge.* When a filter raises win rate but lowers expectancy, you have found a filter that trades frequency for size. That trade is only good if the size reduction is larger than the frequency gain — usually it is not.\n\n*Run the test on out-of-sample data.* A 90-day window with 24 symbols is enough to see a sign flip, not enough to trust the magnitude. Re-run on a held-out period and check whether the rejected-slice expectancy keeps the same sign.\n\n## Closing\n\nFilters that feel safe are often filters that select for late entries. Measure what they reject, not just what they keep.\n\nMore notes like this ship every week on this site.\n\n---\n\n## Daily Picks\n\n\u003e The following pairs are selected from the multi-timeframe trend scanner (Gate.io futures) and are for technical-analysis study only — not investment advice.\n\u003e Data updated: 2026-10-27 08:46:47\n\n### Short\n\n| Pair | Signal | Price | Take Profit | Stop Loss | R/R |\n|------|--------|-------|-------------|-----------|-----|\n| Q | 热点跟涨 | $0.2008 | $0.1848 | $0.2079 | 1:2.3 |\n\n_1 picks selected. Scanner runs every 15 minutes.
","positive_reactions_count":1,"cover_image":null,"tag_list":["pullback","filterdesign","expectancy","abtest"],"canonical_url":"https://www.hkcode.dpdns.org/en/posts/29.html","reading_time_minutes":6,"user":{"name":"mattleeee","username":"mattleeee","twitter_username":null,"github_username":"mattleeee","user_id":4160029,"website_url":null,"profile_image":"https://media2.dev.to/dynamic/image/width=640,height=640,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png","profile_image_90":"https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png"}},{"type_of":"article","id":4918097,"title":"Three-Layer Validation Before You Trust a Backtest","description":"Why One Good Backtest Means Nothing A single equity curve is a sample of size one....","published":true,"published_at":"2026-10-26T01:02:11.123Z","slug":"three-layer-validation-before-you-trust-a-backtest-1hjj","path":"/mattleeee/three-layer-validation-before-you-trust-a-backtest-1hjj","url":"https://dev.to/mattleeee/three-layer-validation-before-you-trust-a-backtest-1hjj","comments_count":0,"public_reactions_count":0,"page_views_count":60,"published_timestamp":"2026-10-26T01:02:11Z","body_markdown":"## Why One Good Backtest Means Nothing\n\nA single equity curve is a sample of size one. Optimize hard enough over enough combinations and something will look great by luck alone. The question is not \"does this rule make money in the backtest?\" but \"does this rule make money for reasons that survive contact with slightly different assumptions?\"\n\nThe gate I use before any rule reaches the live scanner has three layers: a parameter sweep, a monthly consistency check, and a component ablation. Each layer answers a different question, and each has a hard pass/fail threshold. A rule has to clear all three. This post walks through the mechanics, then through a real rejection: a MACD divergence entry that passed layer 1 and died on layers 2 and 3.\n\n## Layer 1: Parameter Sweep\n\n### The question it answers\n\nIs the edge a plateau or a spike? If performance only exists at fast=12, slow=26, signal=9, you have found noise, not a rule. Neighboring parameter values should produce comparable results, and the chosen point should sit near the center of a profitable region, not on its edge.\n\n### The metrics\n\nFor each parameter combination in the grid, record:\n\n- Trade count — anything under ~50 is too thin to say much.\n- Profit factor (PF) — gross wins / gross losses.\n- Expectancy per trade — in R units (risk multiples), so results are comparable across parameter sets.\n- Max drawdown — in R, not percent, so it doesn't depend on position sizing.\n- Plateau score — the fraction of neighboring combinations (within one step in each dimension) that stay above the PF threshold.\n\n### The thresholds\n\n- Median PF across the sweep ≥ 1.3\n- At least 70% of combinations with ≥ 50 trades have PF \u003e 1.0\n- Plateau score at the chosen point ≥ 0.6\n- Chosen point is not on the grid boundary\n\n### The code\n\n

python\nimport itertools\nimport numpy as np\nimport pandas as pd\n\ndef sweep(params_grid, run_backtest, min_trades=50):\n rows = []\n for combo in itertools.product(*params_grid.values()):\n params = dict(zip(params_grid.keys(), combo))\n res = run_backtest(**params)\n if res[\"trades\"] \u003c min_trades:\n continue\n rows.append({**params, **res})\n return pd.DataFrame(rows)\n\ndef plateau_score(df, param_names, chosen, pf_col=\"pf\", threshold=1.0):\n \"\"\"Fraction of 1-step neighbors of `chosen` with PF above threshold.\"\"\"\n neighbors = []\n for name in param_names:\n values = sorted(df[name].unique())\n idx = values.index(chosen[name])\n for offset in (-1, 1):\n j = idx + offset\n if 0 \u003c= j \u003c len(values):\n mask = pd.Series(True, index=df.index)\n for n in param_names:\n target = chosen[n] if n != name else values[j]\n mask \u0026= df[n] == target\n if mask.any():\n neighbors.append(df.loc[mask, pf_col].iloc[0])\n if not neighbors:\n return 0.0\n return float(np.mean(np.array(neighbors) \u003e threshold))\n\ngrid = {\n \"fast\": [8, 10, 12, 14, 16],\n \"slow\": [20, 24, 26, 30, 34],\n \"signal\": [7, 9, 11],\n}\nresults = sweep(grid, run_macd_divergence_backtest)\n\nchosen = {\"fast\": 12, \"slow\": 26, \"signal\": 9}\nscore = plateau_score(results, list(grid.keys()), chosen)\n\nprint(f\"median PF: {results['pf'].median():.2f}\")\nprint(f\"share PF\u003e1: {(results['pf'] \u003e 1.0).mean():.0%}\")\nprint(f\"plateau score at chosen point: {score:.2f}\")\n

\n\nA plateau score of 0.6 means most one-step neighbors still clear PF 1.0. That is a low bar, deliberately. If a rule can't survive a one-step nudge, it has no business being tested further.\n\n## Layer 2: Monthly Consistency\n\n### The question it answers\n\nIs the annual result distributed across time, or is it one monster month doing all the work? A strategy with one +40% month and eleven flat months is not a strategy. It's a lottery ticket that already paid out, and the next ticket is unlikely to pay again.\n\n### The metrics\n\nBucket trades by calendar month and compute:\n\n- Share of profitable months — count of months with positive PnL / total months.\n- Best-month contribution — best month's PnL / total PnL.\n- Top-3-month contribution — same, for the three best months.\n- Median monthly return and worst month in R.\n\n### The thresholds\n\n- ≥ 55% of months positive\n- Best month ≤ 30% of total PnL\n- Top 3 months ≤ 55% of total PnL\n- Worst month no worse than −3R (scaled to your per-trade risk)\n\nThe best-month cap is the important one. It is the metric that catches curve-fit strategies that happen to overlap a single volatility event.\n\n### The code\n\n

python\ndef monthly_consistency(trades: pd.DataFrame, risk_per_trade=1.0):\n \"\"\"trades: columns ['exit_time', 'pnl_r']\"\"\"\n monthly = (trades\n .assign(month=trades[\"exit_time\"].dt.to_period(\"M\"))\n .groupby(\"month\")[\"pnl_r\"]\n .sum()\n .sort_index())\n total = monthly.sum()\n if total \u003c= 0:\n return {\"pass\": False, \"reason\": \"non-positive total PnL\"}\n\n best = monthly.max()\n top3 = monthly.nlargest(3).sum()\n stats = {\n \"months\": len(monthly),\n \"positive_share\": (monthly \u003e 0).mean(),\n \"best_share\": best / total,\n \"top3_share\": top3 / total,\n \"median_month\": monthly.median(),\n \"worst_month\": monthly.min(),\n }\n stats[\"pass\"] = (\n stats[\"months\"] \u003e= 24\n and stats[\"positive_share\"] \u003e= 0.55\n and stats[\"best_share\"] \u003c= 0.30\n and stats[\"top3_share\"] \u003c= 0.55\n and stats[\"worst_month\"] \u003e= -3 * risk_per_trade\n )\n return stats\n

\n\nThe months \u003e= 24 requirement matters. With twelve months, \"positive share\" is dominated by a handful of observations and the test is close to meaningless.\n\n## Layer 3: Robustness Ablation\n\n### The question it answers\n\nDoes each component of the rule contribute, and does the rule degrade gracefully when a component is removed? A robust rule loses some edge when you strip a filter. A fragile rule collapses to zero, or worse, only works with every component present — a sign the components were fitted to each other.\n\n### The metrics\n\nFor each component, run the backtest with that component removed (or replaced by a neutral version) and record:\n\n- PF delta — component PF minus full-rule PF.\n- Expectancy delta in R.\n- Retention — component expectancy / full-rule expectancy.\n\n### The thresholds\n\n- Removing any single component leaves expectancy ≥ 60% of the full rule (graceful degradation).\n- Removing the core component (the entry trigger itself, not a filter) drops expectancy below 50% and PF below 1.1 (the core actually matters).\n- No component removal improves expectancy by more than 15% (if it does, that component is hurting and should be cut).\n\nThe third criterion is the one people skip. It catches filters that were added to patch a specific losing streak in the sample and do nothing out of sample.\n\n### The code\n\n

python\ndef ablation(full_result, variants: dict, core=\"divergence\"):\n \"\"\"variants: name -\u003e backtest result dict with 'pf' and 'expectancy_r'\"\"\"\n base_pf = full_result[\"pf\"]\n base_exp = full_result[\"expectancy_r\"]\n rows = []\n for name, res in variants.items():\n rows.append({\n \"component\": name,\n \"pf\": res[\"pf\"],\n \"expectancy_r\": res[\"expectancy_r\"],\n \"retention\": res[\"expectancy_r\"] / base_exp if base_exp else 0.0,\n \"pf_delta\": res[\"pf\"] - base_pf,\n })\n df = pd.DataFrame(rows)\n\n checks = {\n \"graceful\": (df.loc[df[\"component\"] != core, \"retention\"] \u003e= 0.60).all(),\n \"core_matters\": df.loc[df[\"component\"] == core, \"retention\"].lt(0.50).all(),\n \"no_harmful\": (df[\"retention\"] \u003c= 1.15).all(),\n }\n return df, checks\n

\n\n## The Case Study: A MACD Divergence Entry\n\nThe rule: enter long when price makes a lower low but MACD histogram makes a higher low (bullish divergence), confirmed by a close above the signal line. Exit on a fixed 2R target or a time stop of 20 bars. Filters included a trend filter (price above the 200-EMA), a volume filter (volume above its 20-bar average), and a volatility floor (ATR above its 20-bar median).\n\n### Layer 1: passed\n\nThe sweep over fast ∈ [8,16], slow ∈ [20,34], signal ∈ [7,11] produced 75 combinations, 68 with enough trades. Median PF was 1.41, 78% of combinations had PF above 1.0, and the plateau score at (12, 26, 9) was 0.71. The chosen point sat comfortably inside the grid. Nothing suspicious yet.\n\n### Layer 2: failed\n\nOver 31 months, only 48% of months were positive. Worse, the best month contributed 44% of total PnL, and the top three months contributed 71%. The best month coincided with a broad volatility expansion — the rule caught one regime and did nothing in the other thirty. Median monthly PnL was slightly negative.\n\nVerdict: fail on positive_share (0.48 \u003c 0.55), best_share (0.44 \u003e 0.30), and top3_share (0.71 \u003e 0.55). This alone is enough to reject the rule.\n\n### Layer 3: failed\n\nThe ablation was run anyway, out of curiosity:\n\n| Removed | PF | Expectancy (R) | Retention |\n|---|---|---|---|\n| Trend filter | 1.38 | 0.11 | 0.73 |\n| Volume filter | 1.52 | 0.17 | 1.13 |\n| Volatility floor | 1.09 | 0.06 | 0.40 |\n| Divergence (core) | 0.94 | −0.01 | −0.07 |\n\nTwo problems. The volatility floor is doing most of the work — removing it drops retention to 0.40, well below the 0.60 floor. That means the \"divergence\" rule is largely a low-volatility regime filter wearing a costume. And removing the volume filter improves expectancy by 13%, which is under the 15% harm threshold but close enough to be a warning sign: the filter was probably added to patch a specific stretch of the sample.\n\nThe core divergence component does matter (removal sends expectancy negative), but that doesn't save the rule. A rule whose edge lives mostly in a regime filter, and whose only positive months come from one volatility event, is not a rule. It's a description of one period.\n\n### The rejection\n\nThree-layer result: pass, fail, fail. The rule went into the rejected list with the metrics recorded, so future me doesn't re-test it in six months and get excited all over again. That bookkeeping is half the value of the process.\n\n## Practical Notes\n\n- Run the layers in order. Layer 1 is cheap and filters out most junk. Layer 2 is cheaper than layer 3 and catches the most common failure mode. Don't run ablations on rules that can't pass a monthly check.\n- Log every metric, including failures. A rejection is only useful if it's documented. Keep the sweep DataFrame and the ablation table next to the rule definition.\n- Thresholds are conventions, not laws. The numbers above are tuned to my trade frequency and risk unit. If your rules trade twenty times a year, months \u003e= 24 is too strict; if they trade daily, best_share \u003c= 0.30 is probably too loose. Pick thresholds once, write them down, and don't move them after seeing a result you like.\n- The ablation is the most informative layer. It tells you why a rule works, not just whether it did. When a rule fails layer 3, you often learn something reusable about the market — in this case, that low-volatility regimes are where this class of divergence entry lives, which is a hypothesis worth testing on its own.\n\nMore notes like this ship every week on this site.","positive_reactions_count":0,"cover_image":null,"tag_list":["backtesting","overfitting","parametersweep","robustness"],"canonical_url":"https://www.hkcode.dpdns.org/en/posts/28.html","reading_time_minutes":7,"user":{"name":"mattleeee","username":"mattleeee","twitter_username":null,"github_username":"mattleeee","user_id":4160029,"website_url":null,"profile_image":"https://media2.dev.to/dynamic/image/width=640,height=640,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png","profile_image_90":"https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png"}},{"type_of":"article","id":4912879,"title":"Enforcing HTTPS on GitHub Pages After a Domain Switch","description":"Pointing a new domain at GitHub Pages works on the first try often enough that the greyed-out...","published":true,"published_at":"2026-10-25T01:02:24.885Z","slug":"enforcing-https-on-github-pages-after-a-domain-switch-135","path":"/mattleeee/enforcing-https-on-github-pages-after-a-domain-switch-135","url":"https://dev.to/mattleeee/enforcing-https-on-github-pages-after-a-domain-switch-135","comments_count":0,"public_reactions_count":0,"page_views_count":70,"published_timestamp":"2026-10-25T01:02:24Z","body_markdown":"Pointing a new domain at GitHub Pages works on the first try often enough that the greyed-out checkbox comes as a surprise. You add the custom domain, GitHub accepts it, the site loads over plain HTTP, and the Enforce HTTPS option sits there disabled with a note that the certificate has not been provisioned yet. In most cases nothing is wrong. GitHub is still in the middle of issuing the certificate, and the toggle unlocks on its own once that finishes.\n\nThis post covers the whole lifecycle: what happens after you save a custom domain, how to watch it progress from the command line, why you should fix mixed content before you enforce HTTPS, and what to check when provisioning genuinely fails.\n\n## The provisioning sequence\n\nWhen you save a custom domain for a GitHub Pages site, GitHub runs a fixed pipeline in the background. The UI gives you one checkbox as feedback, but three distinct things happen.\n\n### 1. DNS check\n\nGitHub resolves the domain and confirms it points where it should:\n\n- For an apex domain (example.com), A records must point to the Pages IPs, or an ALIAS/ANAME record must be configured.\n- For a subdomain (www.example.com, blog.example.com), a CNAME record must point to \u003cuser\u003e.github.io (or \u003corg\u003e.github.io).\n\nIf DNS does not resolve correctly at this stage, the certificate request never starts. The UI will keep saying the certificate is pending, and it will keep saying that indefinitely. This is the failure mode worth knowing about, because it looks identical to normal waiting.\n\n### 2. Certificate issuance\n\nOnce DNS checks out, GitHub requests a certificate for the domain (and the www variant where applicable) from its CA. This is the part that takes time. Typical latency is 10 to 30 minutes, but it depends on DNS TTLs and CA queueing, so an hour is not unheard of after a domain switch. During this window the site is reachable over HTTP and, in many cases, already over HTTPS with a certificate that browsers may or may not accept yet.\n\n### 3. Enforce toggle unlocks\n\nOnly after the certificate is live does GitHub enable the Enforce HTTPS checkbox. Ticking it makes Pages redirect all HTTP requests to HTTPS with a 301. Until then, the box stays disabled, and there is no override, no query parameter, no API call that forces it early.\n\nThe practical takeaway: after a domain switch, expect a wait. Do not start changing DNS records while the certificate is pending, because you will reset the process.\n\n## Watching progress from the terminal\n\nThe GitHub settings page is a poor progress indicator. It shows a static message until it flips. curl tells you more.\n\n### Check what is actually being served\n\n

bash\ncurl -I http://example.com\ncurl -I https://example.com\n

\n\nRun both. Before the certificate is live, the HTTPS request typically fails at the TLS handshake, which curl reports as:\n\n

plaintext\ncurl: (60) SSL certificate problem: unable to get local issuer certificate\n

\n\nor a certificate mismatch error. The HTTP request, meanwhile, returns 200 with no redirect. That combination means: DNS is fine, certificate is not ready.\n\nOnce provisioning completes, https:// returns 200 and http:// still returns 200 until you tick the box. After enforcing, http:// returns:\n\n

http\nHTTP/1.1 301 Moved Permanently\nLocation: https://example.com/\n

\n\n### Inspect the served certificate\n\ncurl -I only tells you whether TLS works. To see which certificate is being served and for which names, use openssl:\n\n

bash\necho | openssl s_client -servername example.com -connect example.com:443 2\u003e/dev/null \\\n | openssl x509 -noout -subject -issuer -dates -ext subjectAltName\n

\n\nA healthy result looks roughly like:\n\n

plaintext\nsubject=CN = example.com\nissuer=C = US, O = Let's Encrypt, CN = R3\nnotBefore=...\nnotAfter=...\nX509v3 Subject Alternative Name:\n DNS:example.com, DNS:www.example.com\n

\n\nTwo things to verify here. First, the SAN list must include every hostname you actually use. If you serve www.example.com but the certificate only covers the apex, browsers will warn on the www host even though the apex is fine. Second, check notBefore and notAfter to rule out a stale or expired certificate being served from an edge cache.\n\nIf openssl returns a certificate whose subject is *.github.io or belongs to a different domain, the request is landing on the wrong vhost. That usually means DNS is still pointing at an old host, or a proxy (Cloudflare, for example) is terminating TLS in front of Pages. Cloudflare's \"Full\" or \"Flexible\" SSL modes in front of GitHub Pages are a common source of confusing certificate output.\n\n### Scripting the wait\n\nIf you switch domains often, a small polling script saves a lot of tab-refreshing. This one checks the HTTPS handshake and the redirect status until both are correct, or gives up after a timeout:\n\n

python\nimport ssl\nimport socket\nimport time\nimport urllib.request\nimport urllib.error\n\nDOMAIN = \"example.com\"\nTIMEOUT_MINUTES = 45\nINTERVAL_SECONDS = 120\n\n\ndef tls_ok(host: str, port: int = 443) -\u003e bool:\n ctx = ssl.create_default_context()\n try:\n with socket.create_connection((host, port), timeout=10) as sock:\n with ctx.wrap_socket(sock, server_hostname=host) as tls:\n cert = tls.getpeercert()\n sans = [v for k, v in cert.get(\"subjectAltName\", ()) if k == \"DNS\"]\n return host in sans or f\"*.{host.split('.', 1)[1]}\" in sans\n except (ssl.SSLError, socket.error, OSError):\n return False\n\n\ndef redirects_to_https(url: str) -\u003e bool:\n req = urllib.request.Request(url, method=\"HEAD\")\n try:\n with urllib.request.urlopen(req, timeout=10) as resp:\n return resp.geturl().startswith(\"https://\")\n except urllib.error.HTTPError as e:\n return e.code in (301, 308) and e.headers.get(\"Location\", \"\").startswith(\"https://\")\n except urllib.error.URLError:\n return False\n\n\ndeadline = time.time() + TIMEOUT_MINUTES * 60\nwhile time.time() \u003c deadline:\n cert = tls_ok(DOMAIN)\n enforced = redirects_to_https(f\"http://{DOMAIN}\")\n print(f\"cert={cert} enforced={enforced}\")\n if cert and enforced:\n print(\"Done.\")\n break\n time.sleep(INTERVAL_SECONDS)\nelse:\n print(\"Timed out. Check DNS and the Pages settings.\")\n

\n\nThe tls_ok check is stricter than a plain connection test: it validates that the hostname appears in the SAN list, which is the thing that actually matters for browsers. The redirects_to_https check confirms enforcement is active. Run it from a machine that is not behind the same DNS resolver you just changed, or you may be testing a cached view.\n\n## Fix mixed content before you enforce\n\nThis is the step people skip, and it causes the most visible breakage. Enforcing HTTPS makes the page itself load over TLS, but it does not rewrite the URLs inside your HTML, CSS, or JavaScript. Any resource still referenced as http:// becomes mixed content. Modern browsers block active mixed content (scripts, stylesheets, iframes) outright and either block or auto-upgrade passive mixed content (images), depending on the browser and the resource type.\n\nThe result: the page loads, the layout looks wrong, and the console is full of errors. Users blame the certificate; the real cause is a handful of hard-coded http:// URLs.\n\n### Find them before you flip the switch\n\nCrawl your own site over HTTP and grep the rendered HTML for http:// references. A minimal crawler:\n\n

python\nimport re\nfrom urllib.parse import urljoin, urlparse\nfrom collections import deque\n\nimport requests\n\nSTART = \"http://example.com/\"\nSAME_HOST = urlparse(START).netloc\n\n# Ignore URLs that are fine as http:// (rare) or not resources at all.\nIGNORE = re.compile(r\"^(mailto:|tel:|javascript:|#)\")\n\nseen = set()\nqueue = deque([START])\nfindings = []\n\nsession = requests.Session()\nsession.headers[\"User-Agent\"] = \"mixed-content-audit/1.0\"\n\nwhile queue:\n url = queue.popleft()\n if url in seen or urlparse(url).netloc != SAME_HOST:\n continue\n seen.add(url)\n\n try:\n resp = session.get(url, timeout=10)\n except requests.RequestException:\n continue\n if \"text/html\" not in resp.headers.get(\"Content-Type\", \"\"):\n continue\n\n for match in re.finditer(r'(?:src|href|action)\\s*=\\s*[\"\\'](http://[^\"\\']+)', resp.text):\n findings.append((url, match.group(1)))\n\n for match in re.finditer(r'\u003ca\\s[^\u003e]*href\\s*=\\s*[\"\\']([^\"\\'#]+)', resp.text):\n link = urljoin(url, match.group(1))\n if not IGNORE.match(link):\n queue.append(link)\n\nfor page, asset in findings:\n print(f\"{page} -\u003e {asset}\")\nprint(f\"\\n{len(findings)} http:// references across {len(seen)} pages\")\n

\n\nAnything it prints is a candidate for breakage. The fix is mechanical: rewrite to https://, or better, to protocol-relative or root-relative URLs (//cdn.example.com/lib.js or /assets/lib.js) so the scheme follows the page.\n\n### Common offenders\n\n- Hard-coded CDN links in templates, often copied from documentation years ago.\n- Markdown content with embedded images using http:// URLs.\n- Analytics or font snippets pasted from a provider dashboard that predates HTTPS-by-default.\n- srcset and \u003cpicture\u003e sources, which are easy to miss because they are not plain src attributes.\n- CSS url() references to background images or fonts.\n\nThe crawler above only covers HTML attributes. For a thorough pass, also fetch your CSS files and search them for http://, and check any client-side JavaScript that builds URLs at runtime. A Content-Security-Policy header with upgrade-insecure-requests is a useful safety net, but it is a mitigation, not a fix. Do the rewrite first.\n\n## When provisioning actually fails\n\nIf more than a couple of hours pass and curl -I https://example.com still fails the handshake, the pipeline is stuck. Work through these in order.\n\n### Verify the CNAME value exactly\n\nFor a subdomain, the CNAME record must point to the Pages hostname, not to an IP, not to a proxy, and not to a URL with a path. The value is \u003cuser\u003e.github.io. (or \u003corg\u003e.github.io.), with or without the trailing dot depending on your DNS provider. The apex domain is a special case: it cannot be a CNAME, so it needs A records to the documented GitHub Pages IPs, plus a CNAME file in the repository or the custom domain set in settings.\n\n

bash\ndig +short CNAME www.example.com\ndig +short A example.com\ndig +short example.com NS\n

\n\nCheck the NS output too. If the domain is still delegated to your registrar's parking nameservers because you changed nameservers recently, the records you configured in the new DNS provider are not live yet. That alone will stall provisioning.\n\n### Wait out the TTL\n\nIf you changed DNS records within the last 48 hours, resolvers across the internet are still serving cached answers with the old target. GitHub's checker may have queried a resolver that still had the old record and concluded DNS was wrong. There is no way to force GitHub's check; it retries on its own schedule. Lowering the TTL before a planned migration (to 300 seconds, a day or two in advance) makes this window much shorter.\n\nTo see what different resolvers think, query a public resolver directly:\n\n

bash\ndig +short CNAME www.example.com @1.1.1.1\ndig +short CNAME www.example.com @8.8.8.8\n

\n\nIf these disagree, you are still inside the propagation window. Wait.\n\n### Remove and re-add the custom domain\n\nIf DNS is correct across resolvers, the served certificate is wrong or absent, and several hours have passed, remove the custom domain in the Pages settings, save, and add it back. This cancels the stuck request and starts a fresh one. It is safe, but it resets the clock, so only do it once you are confident DNS is right. Re-adding the domain while DNS is still wrong just reproduces the same stuck state.\n\n### Rule out a proxy in front of Pages\n\nIf the domain is behind Cloudflare or a similar proxy, the certificate GitHub issues may never be served, because the proxy terminates TLS with its own certificate. That is not necessarily a problem, but it changes where you look: the openssl output will show the proxy's issuer, and the \"Enforce HTTPS\" toggle in GitHub may stay disabled because GitHub cannot complete its own validation through the proxy. Either set the proxy to DNS-only during provisioning and re-enable it afterwards, or accept that HTTPS is handled at the proxy layer and configure enforcement there instead.\n\n### Check the CNAME file in the repo\n\nIf you set the custom domain through the repository's CNAME file rather than the settings UI, a mismatch between the file contents and the settings value can cause the domain to flap between configured states. Pick one method and stick to it; the settings UI writes the CNAME file for you.\n\n## A short checklist\n\nBefore enforcing HTTPS after a domain switch:\n\n1. Confirm DNS resolves correctly from multiple public resolvers.\n2. Wait for the certificate; verify it with openssl s_client and check the SAN list covers every hostname.\n3. Crawl the site for http:// asset references and rewrite them.\n4. Tick Enforce HTTPS once the checkbox unlocks.\n5. Verify the redirect with curl -I http://example.com and re-run the crawler against the HTTPS origin to confirm nothing regressed.\n\nThe greyed-out checkbox is normal. The failures that matter are the ones where it stays greyed out for hours, and those almost always trace back to DNS.\n\nMore notes like this ship every week on this site.\n\n---\n\n## Daily Picks\n\n\u003e The following pairs are selected from the multi-timeframe trend scanner (Gate.io futures) and are for technical-analysis study only — not investment advice.\n\u003e Data updated: 2026-10-25 08:56:33\n\n### Long\n\n| Pair | Signal | Price | Take Profit | Stop Loss | R/R |\n|------|--------|-------|-------------|-----------|-----|\n| AXIS | 短平快做多 | $0.2484 | $0.2559 | $0.2422 | 1:1.2 |\n| ORDER | 短平快做多 | $0.0566 | $0.0583 | $0.0552 | 1:1.2 |\n\n_2 picks selected. Scanner runs every 15 minutes.","positive_reactions_count":0,"cover_image":null,"tag_list":["https","githubpages","certificate","enforcehttps"],"canonical_url":"https://www.hkcode.dpdns.org/en/posts/27.html","reading_time_minutes":9,"user":{"name":"mattleeee","username":"mattleeee","twitter_username":null,"github_username":"mattleeee","user_id":4160029,"website_url":null,"profile_image":"https://media2.dev.to/dynamic/image/width=640,height=640,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png","profile_image_90":"https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png"}},{"type_of":"article","id":4908103,"title":"Why You Cannot Put a CNAME on the Apex Domain","description":"The zone apex is not like other names The apex of a zone — the bare domain itself,...","published":true,"published_at":"2026-10-24T01:01:45.853Z","slug":"why-you-cannot-put-a-cname-on-the-apex-domain-3am4","path":"/mattleeee/why-you-cannot-put-a-cname-on-the-apex-domain-3am4","url":"https://dev.to/mattleeee/why-you-cannot-put-a-cname-on-the-apex-domain-3am4","comments_count":0,"public_reactions_count":0,"page_views_count":42,"published_timestamp":"2026-10-24T01:01:45Z","body_markdown":"## The zone apex is not like other names\n\nThe apex of a zone — the bare domain itself, hkcode.dpdns.org in this case — is where the DNS machinery for the entire zone lives. It must carry an SOA record and a set of NS records. That single fact is the reason you cannot point the apex at a CNAME, and it is not a limitation of GitHub Pages or of any registrar. It is baked into the DNS standard.\n\nWhen we set up GitHub Pages for a custom domain, GitHub's documentation asks for a CNAME record. The natural reading is \"point the domain at username.github.io.\" For the apex, that request cannot be satisfied literally. The site ends up at www.hkcode.dpdns.org, and the bare name is handled by a redirect or a set of A/AAAA records. Here is why, and how to wire it up without creating a resolver loop or a broken zone.\n\n## Why a CNAME cannot live at the apex\n\n### The problem: a CNAME claims the whole name\n\nRFC 1034, section 3.6.2 is explicit: a CNAME record is an alias for a domain name, and that name must not have any other data. The text says that if a CNAME RR is present at a node, no other data should be present. RFC 2181, section 10.1 reinforces it: a CNAME record cannot coexist with any other record type at the same owner name, with a narrow exception for DNSSEC-related records (RRSIG, NSEC, and similar).\n\nSo the rule is: at a node, either a CNAME, or everything else. Not both.\n\nNow look at the apex. Every zone needs:\n\n- An SOA record, which describes the zone's primary server, serial number, refresh/retry/expire timers, and negative-caching TTL.\n- At least one NS record set, listing the authoritative name servers.\n\nThose records must exist at the apex by definition — the SOA names the zone origin, and the NS set tells the world who is authoritative for it. If you also tried to place a CNAME at the apex, you would be asking the zone to hold a CNAME and an SOA and NS records at the same owner name. That directly violates RFC 1034/2181.\n\n### What happens if a provider lets you try\n\nMany authoritative servers will simply reject the zone file at load time with an error like CNAME and other data. Some permissive servers accept it, and the result is worse: resolvers receive an answer where the CNAME and the SOA/NS coexist. Behavior becomes undefined. Some resolvers will follow the CNAME and discard the SOA; some will treat the zone as lame. Recursive resolvers that cache the CNAME may return it for queries that should have hit the NS set. You end up with intermittent resolution failures, and the failure mode differs by resolver — the hardest kind of bug to diagnose.\n\nThere is also a subtler issue. A CNAME at the apex would alias the entire zone's identity to another name. Queries for the SOA at the apex would be answered with a CNAME, and the resolver would chase it to the target, which has no authority over your zone. The delegation chain breaks.\n\n### The MX and TXT trap\n\nEven if the RFC allowed it, the apex usually carries more than SOA and NS. Email routing records (MX) and domain-verification or SPF/DKIM records (TXT) commonly live at the bare domain. A CNAME at the apex would make all of those illegal too. For hkcode.dpdns.org, any TXT record used to verify ownership with a third-party service would collide with the CNAME. This is why the \"just CNAME the apex\" idea fails beyond the pure DNS mechanics: real zones have real records at the apex.\n\n## What \"flattening\" providers actually do\n\nSeveral DNS providers (Cloudflare, DNSimple, Route 53 with alias records, and others) offer a record type called ALIAS, ANAME, or a \"CNAME flattening\" feature. This is the workaround people reach for, but it is not a CNAME at the apex. It is a server-side behavior.\n\nHere is the mechanism. You create an ALIAS/ANAME record at the apex pointing to a target name. The provider's authoritative servers do not publish a CNAME. Instead, when a query for the apex arrives, the authoritative server:\n\n1. Resolves the target name itself (or uses its cached answer).\n2. Returns the resulting A/AAAA records directly, as if they were A/AAAA records at the apex.\n\nThe zone on disk still has SOA and NS at the apex; the ALIAS is a provider-specific synthetic record. To the outside world, the apex answers with address records, not a CNAME, so the RFC constraint is respected. The provider has effectively flattened the indirection.\n\nThis works, but it is not portable. The ALIAS record type is not standardized in the way A and CNAME are; its semantics differ between providers. If you export the zone and move to a provider that does not support it, the record either disappears or is written out as a literal CNAME, which then breaks the zone. Also, the target's TTL becomes the provider's resolution TTL, not the original CNAME TTL, so you lose some control over caching.\n\nFor GitHub Pages specifically, the target username.github.io resolves to GitHub's CDN addresses. Flattening works, but you are pinning the apex to whatever those addresses are at query time. If GitHub rotates them, your provider's cached resolution handles it, but you are trusting the provider's resolver behavior.\n\n## How GitHub Pages handles www versus apex\n\nGitHub Pages has a specific model for custom domains:\n\n- www subdomain: GitHub recommends a CNAME record pointing www.hkcode.dpdns.org to hkcode.github.io (or username.github.io). This is the canonical setup. GitHub's servers see the Host: www.hkcode.dpdns.org header, match it against the configured custom domain, and serve the site.\n- Apex domain: GitHub documents A records (and AAAA records for IPv6) pointing the apex at GitHub's published IP addresses. These are static addresses that GitHub maintains. The apex is not a CNAME; it is a set of address records.\n\nWhen you configure a custom domain in the repository settings, GitHub stores it and serves requests whose Host header matches. If you set the custom domain to www.hkcode.dpdns.org, requests to the apex hkcode.dpdns.org will not match unless GitHub also handles the redirect.\n\nGitHub Pages does provide an HTTP-level redirect from the apex to the www host when the custom domain is configured as the www name and the apex resolves to GitHub. The redirect is a 301 served by GitHub's front end, not a DNS-level redirect. For that to work, the apex must still resolve to GitHub's servers — via A/AAAA records — so the request reaches GitHub and gets redirected. If the apex has no records at all, the browser never reaches GitHub and there is no redirect.\n\nSo the practical split is:\n\n- DNS for www → CNAME to username.github.io.\n- DNS for apex → A/AAAA records to GitHub's addresses (or a provider redirect).\n- GitHub config → custom domain set to the www name; GitHub 301s the bare name to www.\n\n## The practical setup\n\n### Step 1: CNAME the www subdomain\n\nAt your DNS provider, create:\n\n

plaintext\nwww.hkcode.dpdns.org. 3600 IN CNAME hkcode.github.io.\n

\n\nNo other records at www. If you previously had an A record there, remove it. A CNAME and an A record at the same name is the same violation as at the apex, just at a different node.\n\n### Step 2: Handle the apex\n\nTwo options, depending on your provider.\n\n*Option A — GitHub's A/AAAA records.* Create A records (and AAAA if you want IPv6) at the apex pointing to GitHub's published addresses. These are documented by GitHub and change rarely, but check them before you paste. The apex then answers with addresses, and GitHub's front end 301s to www.\n\n*Option B — provider redirect.* If your provider offers an HTTP redirect service for the apex (some registrars and DNS hosts do), point the apex at that redirect and send it to https://www.hkcode.dpdns.org. This avoids hardcoding GitHub's IPs. The tradeoff is that the redirect is handled by the provider, so you depend on their uptime and their support for HTTPS on the redirect.\n\nDo not use an ALIAS/ANAME at the apex unless you have verified the provider's implementation and are comfortable with the non-portability. It works, but it is a provider feature, not a DNS standard.\n\n### Step 3: Verify with code\n\nDo not trust the control panel. Query the records directly and check the invariants. A small script using dnspython catches the common mistakes: a stray CNAME at the apex, a missing SOA, or a CNAME and A record coexisting at www.\n\n

```python\nimport dns.resolver\nimport dns.exception\n\nAPEX = \"hkcode.dpdns.org\"\nWWW = f\"www.{APEX}\"\n\ndef query(name, rtype):\n try:\n answer = dns.resolver.resolve(name, rtype, lifetime=5)\n return [r.to_text() for r in answer]\n except dns.resolver.NoAnswer:\n return []\n except dns.resolver.NXDOMAIN:\n return [\"NXDOMAIN\"]\n except dns.exception.Timeout:\n return [\"TIMEOUT\"]\n\ndef check_zone(apex, www):\n problems = []\n\n # The apex must have SOA and NS. A CNAME here is illegal.\n apex_cname = query(apex, \"CNAME\")\n if apex_cname and apex_cname != [\"NXDOMAIN\"]:\n problems.append(f\"apex has a CNAME: {apex_cname}\")\n\n if not query(apex, \"SOA\"):\n problems.append(\"apex has no SOA\")\n if not query(apex, \"NS\"):\n problems.append(\"apex has no NS\")\n\n # The www node should be a CNAME and nothing else.\n www_cname = query(www, \"CNAME\")\n www_a = query(www, \"A\")\n if not www_cname:\n problems.append(\"www has no CNAME\")\n if www_a and www_cname:\n problems.append(\"www has both CNAME and A records\")\n\n return problems\n\nif __name
_ == \"main\":\n issues = check_zone(APEX, WWW)\n if issues:\n for issue in issues:\n print(\"FAIL:\", issue)\n else:\n print(\"zone looks consistent\")\n

\n\nRun it against the authoritative servers, not just your local resolver, so you see what the world sees. You can set the resolver explicitly:\n\n

python\nimport dns.resolver\n\nresolver = dns.resolver.Resolver(configure=False)\n# Use the provider's authoritative nameservers, from the NS set.\nresolver.nameservers = [\"ns1.example.net\", \"ns2.example.net\"]\nresolver.lifetime = 5\n

\n\n### Step 4: Check the HTTP redirect\n\nOnce DNS is correct, confirm the redirect behavior with a request that does not follow redirects:\n\n

python\nimport urllib.request\nimport urllib.error\n\ndef check_redirect(url):\n req = urllib.request.Request(url, method=\"HEAD\")\n opener = urllib.request.build_opener(urllib.request.HTTPRedirectHandler)\n try:\n # Prevent automatic following so we can inspect the 301.\n class NoRedirect(urllib.request.HTTPRedirectHandler):\n def redirect_request(self, args, **kwargs):\n return None\n opener = urllib.request.build_opener(NoRedirect)\n resp = opener.open(req, timeout=10)\n print(url, \"-\u003e\", resp.status)\n except urllib.error.HTTPError as e:\n location = e.headers.get(\"Location\")\n print(url, \"-\u003e\", e.code, \"Location:\", location)\n except Exception as e:\n print(url, \"-\u003e error:\", e)\n\ncheck_redirect(\"http://hkcode.dpdns.org/\")\ncheck_redirect(\"https://hkcode.dpdns.org/\")\n```

\n\nYou want a 301 (or 308) with Location: https://www.hkcode.dpdns.org/. If you get a connection error, the apex has no address records and the redirect never happens. If you get a 200, GitHub is serving the apex directly — check whether the custom domain in the repo settings is set to the bare name, which changes the canonical host.\n\n## Common failure modes\n\n- **CNAME at the apex \"because the provider allowed it.\"
* The zone loads, but some resolvers fail on the SOA or NS query. Symptom: intermittent SERVFAIL from specific networks.\n- CNAME at www plus an A record \"for safety.\" Same violation, smaller blast radius. Remove the A record.\n- Apex A records but no www CNAME. The site works at the bare name but not at www, or vice versa, and the canonical URL drifts.\n- Provider redirect without HTTPS. The apex redirects over HTTP only, and browsers block or warn. Make sure the redirect target uses https://.\n- Stale TTLs during migration. Lower the TTL on the old records to 300 seconds a day before the change, then raise it after the switch settles.\n\n## Summary\n\nA CNAME at the apex is not a configuration choice; it is a contradiction of the zone's required SOA and NS records under RFC 1034 and 2181. ALIAS/ANAME records are a provider-side flattening trick that returns address records instead of a CNAME, and they work but do not travel. GitHub Pages expects a CNAME on www and either A/AAAA records or a provider redirect on the apex, with an HTTP 301 from the bare name to www. Set the CNAME on the subdomain, give the apex address records or a redirect, and verify both the DNS invariants and the redirect with a script rather than the control panel.\n\nMore notes like this ship every week on this site.\n\n---\n\n## Daily Picks\n\n\u003e The following pairs are selected from the multi-timeframe trend scanner (Gate.io futures) and are for technical-analysis study only — not investment advice.\n\u003e Data updated: 2026-10-23 08:46:52\n\n### Long\n\n| Pair | Signal | Price | Take Profit | Stop Loss | R/R |\n|------|--------|-------|-------------|-----------|-----|\n| BTW | 高质量标的做多 | $1.8281 | $1.9561 | $1.7641 | 1:2.0 |\n\n_1 picks selected. Scanner runs every 15 minutes.","positive_reactions_count":0,"cover_image":null,"tag_list":["apexdomain","cname","rfc","wwwsubdomain"],"canonical_url":"https://www.hkcode.dpdns.org/en/posts/25.html","reading_time_minutes":9,"user":{"name":"mattleeee","username":"mattleeee","twitter_username":null,"github_username":"mattleeee","user_id":4160029,"website_url":null,"profile_image":"https://media2.dev.to/dynamic/image/width=640,height=640,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png","profile_image_90":"https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png"}},{"type_of":"article","id":4908102,"title":"Serving a Bilingual Static Site on GitHub Pages","description":"Why one repo, two sites A Chinese blog already lived at the domain root: example.com/. The...","published":true,"published_at":"2026-10-24T01:01:42.456Z","slug":"serving-a-bilingual-static-site-on-github-pages-3kkj","path":"/mattleeee/serving-a-bilingual-static-site-on-github-pages-3kkj","url":"https://dev.to/mattleeee/serving-a-bilingual-static-site-on-github-pages-3kkj","comments_count":0,"public_reactions_count":0,"page_views_count":30,"published_timestamp":"2026-10-24T01:01:42Z","body_markdown":"## Why one repo, two sites\n\nA Chinese blog already lived at the domain root: example.com/. The goal was an English sibling at example.com/en/ without a second repository, a second domain, or a second deploy pipeline. That constraint drives every decision below.\n\nThe layout that ended up working:\n\n

plaintext\nrepo/\n├── CNAME # single domain for both languages\n├── published/ # Chinese article manifests (one JSON per post)\n├── published_en/ # English article manifests\n├── templates/ # shared design system, Jinja2\n├── static/ # shared CSS, JS, fonts\n├── build.py # generator for the root site\n├── build_en.py # generator for /en\n└── docs/ # GitHub Pages output (root = Chinese, /en = English)\n ├── index.html\n ├── sitemap.xml\n ├── rss.xml\n └── en/\n ├── index.html\n ├── sitemap.xml\n └── rss.xml\n

\n\nGitHub Pages serves docs/ from the branch, so the directory tree is the URL tree. The Chinese site is the root; the English site is a subdirectory. One CNAME file, one certificate, one git push.\n\n## Separate manifests prevent cross-language idempotency bugs\n\nThe first version kept one manifest directory with a lang field per article. Generation was supposed to be idempotent: run build.py twice, get identical output. It wasn't. The bug came from how the generator picked slugs and resolved \"previous/next\" links.\n\nWith a single pool, the Chinese generator saw English articles as candidates for adjacency and for slug-collision checks. A Chinese post at 2024/03/foo and an English post at 2024/03/foo (same slug, different language) would race for the same output path. Whichever ran second silently overwrote the first. Worse, rerunning the build could change which one won if the manifest glob order shifted. Non-deterministic output from deterministic input is the worst class of build bug: it looks fine locally, then a post disappears in production a week later.\n\nThe fix is boring and total: two manifest directories, published/ and published_en/, and each generator reads only its own. No lang filter, no shared slug namespace, no cross-language adjacency. The English generator cannot see Chinese manifests, so it cannot be affected by them.\n\n

python\nfrom pathlib import Path\nimport json\n\nMANIFEST_DIRS = {\n \"zh\": Path(\"published\"),\n \"en\": Path(\"published_en\"),\n}\n\ndef load_manifests(lang: str) -\u003e list[dict]:\n \"\"\"Load only this language's manifests, sorted newest first.\"\"\"\n posts = []\n for path in MANIFEST_DIRS[lang].glob(\"*.json\"):\n with path.open(encoding=\"utf-8\") as f:\n post = json.load(f)\n post[\"_source\"] = path.name\n posts.append(post)\n posts.sort(key=lambda p: p[\"date\"], reverse=True)\n\n slugs = [p[\"slug\"] for p in posts]\n if len(slugs) != len(set(slugs)):\n raise ValueError(f\"duplicate slug in {lang} manifests\")\n\n return posts\n

\n\nThe duplicate-slug check now guards a single language, which is exactly the invariant you want: slugs must be unique within a site, not across sites, because the two sites live at different URL prefixes.\n\nThe generators share the design system but nothing else. build.py and build_en.py both import templates/ and static/, both write into docs/, but each owns its manifest set and its own URL prefix. If the English site needs a different layout tweak later, it can diverge without touching the Chinese build.\n\n## One prefix constant keeps links consistent\n\nEvery internal link needs two pieces: the site root (\"\" for Chinese, \"en\" for English) and the article path. Hardcoding either one is how you end up with an English page linking to a Chinese article, or a link that works locally (/en/...) but 404s under a project-page base path.\n\nThe fix is a single constant per build plus a helper that joins them. Nothing else in the codebase constructs internal URLs by string concatenation.\n\n

python\n# In build.py\nSITE_PREFIX = \"\" # Chinese site at the domain root\nLANG = \"zh\"\n\n# In build_en.py\nSITE_PREFIX = \"en\" # English site under /en\nLANG = \"en\"\n

\n\nBoth files import one shared module for URL construction:\n\n

python\n# urls.py\ndef site_url(prefix: str, path: str = \"\") -\u003e str:\n \"\"\"Absolute-path URL for an internal page.\n\n site_url(\"\", \"about/\") -\u003e \"/about/\"\n site_url(\"en\", \"about/\") -\u003e \"/en/about/\"\n \"\"\"\n parts = [p.strip(\"/\") for p in (prefix, path) if p.strip(\"/\")]\n return \"/\" + \"/\".join(parts) + (\"/\" if path.endswith(\"/\") or not path else \"\")\n

\n\nThen the templates never see raw paths. The build passes a url function into the Jinja2 environment:\n\n

python\nfrom jinja2 import Environment, FileSystemLoader\nfrom urls import site_url\n\ndef make_env(prefix: str) -\u003e Environment:\n env = Environment(loader=FileSystemLoader(\"templates\"), autoescape=True)\n env.globals[\"url\"] = lambda path=\"\": site_url(prefix, path)\n return env\n

\n\nIn templates, links become {{ url('posts/' + post.slug + '/') }} and assets become {{ url('static/main.css') }}. Because url closes over the prefix, the same template renders correct links for both languages. A Chinese page emits /posts/foo/; an English page emits /en/posts/foo/. There is no branch in the template and no opportunity to forget the prefix.\n\n### The asset trap\n\nShared assets under static/ are the one place where a prefix-relative link is wrong. If the English page links to /en/static/main.css but the file only exists at /static/main.css, you get a 404 on every English page. Two options: emit absolute asset URLs from the domain root, or mirror the assets under both prefixes. Mirroring doubles the files and the cache entries. The cleaner choice is a separate helper for assets that ignores the language prefix:\n\n

python\nenv.globals[\"asset\"] = lambda path: \"/static/\" + path.lstrip(\"/\")\n

\n\nTemplates use {{ asset('main.css') }} for CSS/JS/fonts and {{ url(...) }} for anything that belongs to a language. One rule, applied consistently, and the asset 404s stop.\n\n## Cross-language navigation in both navs\n\nEach nav needs a link to the equivalent page in the other language, not just the other homepage. On an article page, \"English\" should go to that article's English version if it exists; otherwise fall back to the English index.\n\nThat requires a mapping between the two manifest sets. The simplest reliable key is an explicit translation_key field present in both manifests for paired articles:\n\n

python\ndef build_translation_map(zh_posts, en_posts):\n \"\"\"Map (lang, translation_key) -\u003e slug for cross-language links.\"\"\"\n mapping = {}\n for lang, posts in ((\"zh\", zh_posts), (\"en\", en_posts)):\n for post in posts:\n key = post.get(\"translation_key\")\n if key:\n mapping[(lang, key)] = post[\"slug\"]\n return mapping\n

\n\nWhen rendering a Chinese post with translation_key: \"gh-pages-bilingual\", the nav looks up (\"en\", \"gh-pages-bilingual\"). If found, the link points to /en/posts/\u003cslug\u003e/; if not, it points to /en/. Same logic in reverse for the English site. Articles without a translation still get a working language switch, just at the section root.\n\n

python\ndef alt_lang_href(post, mapping, target_lang, target_prefix):\n key = post.get(\"translation_key\")\n slug = mapping.get((target_lang, key)) if key else None\n if slug:\n return site_url(target_prefix, f\"posts/{slug}/\")\n return site_url(target_prefix)\n

\n\nThe hreflang tags in \u003chead\u003e use the same function, so the machine-readable and human-readable links can't drift apart:\n\n

python\ndef hreflang_links(post, mapping):\n links = [(\"zh\", site_url(\"\", f\"posts/{post['slug']}/\") if post[\"lang\"] == \"zh\"\n else site_url(\"\", f\"posts/{mapping[('zh', post['translation_key'])]}/\"))]\n # ... same for en, plus x-default pointing at the Chinese page\n

\n\nIn practice it's cleaner to compute both URLs once and pass a small dict into the template, but the principle holds: cross-language links are derived from the translation map, never typed by hand.\n\n## Per-language sitemaps and feeds\n\nA single root sitemap.xml listing both languages is legal, but it makes Search Console reporting muddy and forces every URL to carry hreflang annotations in one file. Since the two sites already have separate manifests and separate generators, each generator writes its own sitemap and feed under its own prefix:\n\n- docs/sitemap.xml and docs/rss.xml — Chinese\n- docs/en/sitemap.xml and docs/en/rss.xml — English\n\n

python\nfrom xml.sax.saxutils import escape\nfrom datetime import datetime\n\ndef write_sitemap(posts, prefix: str, out_path: Path, base: str = \"https://example.com\"):\n urls = [f\"{base}{site_url(prefix)}\"]\n urls += [f\"{base}{site_url(prefix, f'posts/{p[\\\"slug\\\"]}/')}\" for p in posts]\n\n body = \"\\n\".join(\n f\" \u003curl\u003e\u003cloc\u003e{escape(u)}\u003c/loc\u003e\u003c/url\u003e\" for u in urls\n )\n xml = (\n '\u003c?xml version=\"1.0\" encoding=\"UTF-8\"?\u003e\\n'\n '\u003curlset xmlns=\"http://www.sitemaps.org/schemas/sitemap/0.9\"\u003e\\n'\n f\"{body}\\n\u003c/urlset\u003e\\n\"\n )\n out_path.write_text(xml, encoding=\"utf-8\")\n

\n\nThe RSS feed follows the same shape, reading \u003clink\u003e values from site_url(prefix, ...). Because both use the same prefix constant, a feed can never contain a URL from the wrong language.\n\nIf you want the sitemaps discoverable without submitting each one manually, add a sitemap index at the root:\n\n

python\ndef write_sitemap_index(base: str, out_path: Path):\n entries = [\n f\"{base}/sitemap.xml\",\n f\"{base}/en/sitemap.xml\",\n ]\n body = \"\\n\".join(\n f\" \u003csitemap\u003e\u003cloc\u003e{escape(u)}\u003c/loc\u003e\u003c/sitemap\u003e\" for u in entries\n )\n out_path.write_text(\n '\u003c?xml version=\"1.0\" encoding=\"UTF-8\"?\u003e\\n'\n '\u003csitemapindex xmlns=\"http://www.sitemaps.org/schemas/sitemap/0.9\"\u003e\\n'\n f\"{body}\\n\u003c/sitemapindex\u003e\\n\",\n encoding=\"utf-8\",\n )\n

\n\n## After the domain switch: submit the second sitemap\n\nThe domain move is where this layout earns its keep. When the site moved to a custom domain, the CNAME file changed once and both languages moved together. GitHub Pages issued the certificate for the new domain, and the root site's URLs redirected cleanly because they were always relative to the domain root.\n\nThe English site needed one extra step. Search Console properties are per-domain (or per-URL-prefix), and the new domain was verified fresh. The root sitemap.xml was submitted immediately, but /en/sitemap.xml sat unsubmitted for a week. English pages were indexed slowly and with stale titles. The fix was to open the new property's Sitemaps report and add https://example.com/en/sitemap.xml as a second sitemap. Within days the English pages were crawled at the same rate as the Chinese ones.\n\nTwo practical notes from that episode. First, submit the sitemap index if you wrote one; Search Console will follow it. Second, the hreflang annotations matter more after a domain change, because the crawler is re-evaluating which URLs are alternates of which. If your language switch links were broken by the move, the annotations would point at dead URLs and the English pages would be treated as orphans. Keeping the language switch derived from the translation map, as above, means a domain change doesn't touch link logic at all.\n\n## What the layout buys you\n\nThe whole design rests on three invariants: one manifest directory per language, one prefix constant per build, one URL helper used everywhere. Everything else — sitemaps, feeds, hreflang, cross-language nav — falls out of those. The build is idempotent because the two generators can't see each other's inputs. Links are consistent because no template constructs a URL by hand. And adding a third language later is a matter of a new manifest directory, a new prefix, and a new generator file, not a refactor.\n\nMore notes like this ship every week on this site.\n\n---\n\n## Daily Picks\n\n\u003e The following pairs are selected from the multi-timeframe trend scanner (Gate.io futures) and are for technical-analysis study only — not investment advice.\n\u003e Data updated: 2026-10-24 08:47:12\n\n### Long\n\n| Pair | Signal | Price | Take Profit | Stop Loss | R/R |\n|------|--------|-------|-------------|-----------|-----|\n| BIRB | 弱做多 | $0.1706 | $0.1835 | $0.1621 | 1:1.5 |\n\n_1 picks selected. Scanner runs every 15 minutes.
","positive_reactions_count":0,"cover_image":null,"tag_list":["multilingualsite","githubpages","directorylayout","sitemap"],"canonical_url":"https://www.hkcode.dpdns.org/en/posts/26.html","reading_time_minutes":7,"user":{"name":"mattleeee","username":"mattleeee","twitter_username":null,"github_username":"mattleeee","user_id":4160029,"website_url":null,"profile_image":"https://media2.dev.to/dynamic/image/width=640,height=640,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png","profile_image_90":"https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png"}},{"type_of":"article","id":4892568,"title":"The CNAME Trailing Dot: One Character That Broke the Domain","description":"The CNAME Trailing Dot: One Character That Broke the Domain The migration was supposed to...","published":true,"published_at":"2026-10-22T01:01:29.620Z","slug":"the-cname-trailing-dot-one-character-that-broke-the-domain-2aeo","path":"/mattleeee/the-cname-trailing-dot-one-character-that-broke-the-domain-2aeo","url":"https://dev.to/mattleeee/the-cname-trailing-dot-one-character-that-broke-the-domain-2aeo","comments_count":0,"public_reactions_count":0,"page_views_count":30,"published_timestamp":"2026-10-22T01:01:29Z","body_markdown":"## The CNAME Trailing Dot: One Character That Broke the Domain\n\nThe migration was supposed to take ten minutes. New domain registered on a free-tier registrar, GitHub Pages already serving the site on mattlee.github.io, so all that remained was a CNAME record pointing www.example.dev at the GitHub Pages hostname. I typed mattlee.github.io into the panel's value field, saved, and waited. Then I waited some more.\n\ndig returned NXDOMAIN for the apex and SERVFAIL for www. GitHub Pages reported no custom domain configured. The DNS panel insisted the record existed, and it did — just not the record anyone intended to create.\n\nThe fix was a single character. The value needed to be mattlee.github.io., with the trailing dot. Without it, the registrar stored a relative name, and the record resolved to mattlee.github.io.example.dev. — a hostname inside my own zone that points at nothing.\n\n### What the trailing dot actually means\n\nDNS has two kinds of names: absolute (fully qualified) and relative. The distinction is encoded in the presentation format by a terminal dot. mattlee.github.io. is an absolute name — it is fully specified and the resolver stops there. mattlee.github.io is a relative name — it is a prefix to be completed by appending an origin.\n\nThis is not a quirk of one vendor. RFC 1034 and RFC 1035 define the master file format with this convention: a domain name that does not end in a dot is interpreted relative to the current origin. In a zone file, the origin is the zone itself. So inside the zone for example.dev, the relative name mattlee.github.io expands to mattlee.github.io.example.dev.\n\nA zone file makes this visible:\n\n

dns\n$ORIGIN example.dev.\nwww IN CNAME mattlee.github.io. ; absolute: mattlee.github.io.\nblog IN CNAME mattlee.github.io ; relative: mattlee.github.io.example.dev.\n

\n\nBoth lines look plausible in a web form. Only the first one does what the author meant. The second creates a CNAME that points back into the same zone, to a name that has no A record. Resolvers chase it, find nothing, and return NXDOMAIN or SERVFAIL depending on how far the chain gets.\n\nWhy does the protocol work this way at all? Because relative names are useful. Zone files are full of them — www, mail, api — and appending the origin keeps configuration short and portable across zones. The trailing dot is the escape hatch that says \"I am done, this is complete.\" The format is terse by design, and the dot is the only signal that distinguishes the two cases.\n\n### Why some panels add the dot and some don't\n\nRegistrar and DNS-provider UIs sit on a spectrum between two philosophies.\n\n*Zone-file faithful.* Some panels treat the value field exactly as a zone file would. If you omit the dot, you get a relative name. These panels are technically correct and operationally hostile — most users typing a CNAME value mean an absolute hostname, because they copied it from a documentation page or a hosting dashboard where the trailing dot is usually omitted.\n\n*Lenient normalizers.* Other panels assume every value is absolute and append the dot for you on save. Many also strip a dot you type in, then re-add it, so the stored record is always fully qualified. These panels hide the problem entirely, which is why the same record that works on one provider fails on another.\n\n*Hybrid.* A third class stores what you type but normalizes on display, so the panel shows mattlee.github.io whether or not the dot is present. You cannot tell from the UI what was actually written to the zone. This is the worst case for debugging, because the panel's appearance actively misleads you.\n\nThe practical consequence: a CNAME value that works on your old registrar may break on your new one, even though both panels display the same string. The record is not the string you see. The record is what the provider serialized into the zone.\n\n### How to verify what was actually stored\n\nNever trust the panel. Query the authoritative nameservers directly and read the ANSWER section.\n\n

bash\ndig www.example.dev CNAME +short\n

\n\nIf the record is correct, you get:\n\n


\n\nNote the trailing dot in the output. `dig` always prints absolute names with the dot, because that is what the wire format contains — DNS messages carry fully qualified names, and the presentation layer adds the dot back on output.\n\nIf the record is broken, you get something like:\n\n

```plaintext\nmattlee.github.io.example.dev.\n```

\n\nThat is the smoking gun. The resolver is telling you the stored name is relative and the origin was appended. If you see the zone name glued onto the end of your intended target, the value field is missing its dot.\n\nYou can also query the authoritative server directly, bypassing caching and any recursive resolver behavior:\n\n

```bash\ndig @ns1.registrar.net www.example.dev CNAME +noall +answer\n```

\n\nThe `+noall +answer` flags trim the output to just the answer section, which is where the CNAME target lives. If the answer section is empty and the authority section shows `NXDOMAIN`, the name may not exist at all — check that the record saved and that the zone is active.\n\n`nslookup` works too, though its output is noisier:\n\n

```bash\nnslookup -type=CNAME www.example.dev ns1.registrar.net\n```

\n\nLook at the `canonical name =` line. Same rule: a trailing dot means absolute, no trailing dot in a zone-file-style answer means relative.\n\nFor a full trace of the resolution chain, including where it fails:\n\n

```bash\ndig www.example.dev +trace\n```

\n\n`+trace` walks from the root down. If the delegation to your zone succeeds but the CNAME target never resolves, the failure is inside your zone — almost always the missing dot.\n\n### Automating the check\n\nOnce you know what to look for, the check is trivial to script. This is worth wiring into your deploy pipeline, because a broken CNAME is invisible until someone tries to load the site.\n\n

```python\nimport subprocess\nimport sys\n\n\ndef get_cname(name: str, server: str | None = None) -\u003e str | None:\n    \"\"\"Return the CNAME target for `name`, or None if no CNAME exists.\"\"\"\n    cmd = [\"dig\", \"+short\", \"CNAME\", name]\n    if server:\n        cmd.insert(1, f\"@{server}\")\n    result = subprocess.run(cmd, capture_output=True, text=True, timeout=10)\n    target = result.stdout.strip()\n    return target or None\n\n\ndef check_cname(name: str, expected: str, server: str | None = None) -\u003e bool:\n    \"\"\"\n    Verify that `name` is a CNAME to `expected`.\n\n    `expected` should be given without a trailing dot; the comparison\n    normalizes both sides so the check is independent of presentation.\n    \"\"\"\n    target = get_cname(name, server)\n    if target is None:\n        print(f\"FAIL: no CNAME record for {name}\")\n        return False\n\n    normalized = target.rstrip(\".\")\n    if normalized != expected.rstrip(\".\"):\n        print(f\"FAIL: {name} -\u003e {target} (expected {expected}.)\")\n        print(\"      If the target ends in your zone name, the value is\")\n        print(\"      relative and is missing its trailing dot.\")\n        return False\n\n    print(f\"OK: {name} -\u003e {target}\")\n    return True\n\n\nif __name__ == \"__main__\":\n    ok = check_cname(\"www.example.dev\", \"mattlee.github.io\")\n    sys.exit(0 if ok else 1)\n```

\n\nThe `rstrip(\".\")` normalization matters: `dig +short` prints the absolute form with a dot, but you will often pass the expected value without one, as copied from a dashboard. Comparing the normalized forms avoids false alarms while still catching the real failure — a target that has your zone name appended.\n\nIf you want to detect the relative-name bug specifically, check whether the target ends with the zone:\n\n

```python\ndef looks_relative(target: str, zone: str) -\u003e bool:\n    \"\"\"Heuristic: a CNAME target that ends in the zone is probably relative.\"\"\"\n    return target.rstrip(\".\").endswith(zone.rstrip(\".\"))\n```

\n\nThis is a heuristic, not a proof — a legitimate CNAME can point into your own zone, and internal service records often do. But for a public web CNAME pointing at an external host, a target that ends in your own zone name is a near-certain sign of a missing dot.\n\n### The general lesson\n\nDNS tools are literal and unapologetic. There is no fuzzy matching, no \"did you mean,\" no warning when you save a relative name where you meant an absolute one. The zone is a text format with a precise grammar, and the provider either follows it or silently normalizes it for you. When it follows it, you get exactly what you typed, including the consequences.\n\nThis bites in predictable places:\n\n- **CNAME and MX values** are the most common victims, because they are the record types where you type a hostname rather than an IP address.\n- **Copy-paste from documentation** usually drops the trailing dot, because docs are written for humans and the dot is noise to most readers.\n- **Provider migration** surfaces latent bugs, because the new panel may interpret the value differently from the old one.\n- **TXT records** are not affected the same way — the content is an opaque string, not a domain name — but SPF and DKIM records that embed domain names inside the string can carry the same confusion.\n\nThe habit that prevents all of it: when a record value is a hostname, type the trailing dot. It is harmless on panels that normalize, and load-bearing on panels that don't. Then verify with `dig` and read the `ANSWER` section, because the panel is a UI and the zone is the truth.\n\nThe domain came up the moment the dot was added. The record had been \"correct\" in the panel for an hour. It had never been correct in the zone.\n\nMore notes like this ship every week on this site.\n\n---\n\n## Daily Picks\n\n\u003e The following pairs are selected from the multi-timeframe trend scanner (Gate.io futures) and are for technical-analysis study only — not investment advice.\n\u003e Data updated: 2026-10-22 08:46:44\n\n### Short\n\n| Pair | Signal | Price | Take Profit | Stop Loss | R/R |\n|------|--------|-------|-------------|-----------|-----|\n| ARIA | 短平快做空 | $0.0655 | $0.0635 | $0.0671 | 1:1.2 |\n\n_1 picks selected. Scanner runs every 15 minutes._","positive_reactions_count":0,"cover_image":null,"tag_list":["cname","trailingdot","fqdn","dnsdebugging"],"canonical_url":"https://www.hkcode.dpdns.org/en/posts/24.html","reading_time_minutes":7,"user":{"name":"mattleeee","username":"mattleeee","twitter_username":null,"github_username":"mattleeee","user_id":4160029,"website_url":null,"profile_image":"https://media2.dev.to/dynamic/image/width=640,height=640,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png","profile_image_90":"https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png"}},{"type_of":"article","id":4886183,"title":"Free Domain Options in 2026: From eu.org to DigitalPlat dpdns.org","description":"The 2026 Free Domain Landscape   If you run static sites for side projects, a domain is the...","published":true,"published_at":"2026-10-21T01:01:48.145Z","slug":"free-domain-options-in-2026-from-euorg-to-digitalplat-dpdnsorg-3k3o","path":"/mattleeee/free-domain-options-in-2026-from-euorg-to-digitalplat-dpdnsorg-3k3o","url":"https://dev.to/mattleeee/free-domain-options-in-2026-from-euorg-to-digitalplat-dpdnsorg-3k3o","comments_count":0,"public_reactions_count":4,"page_views_count":106,"published_timestamp":"2026-10-21T01:01:48Z","body_markdown":"## The 2026 Free Domain Landscape\n\nIf you run static sites for side projects, a domain is the one recurring cost you can usually avoid. The catch is that free domain providers are unstable in a way paid registries are not. Programs open, close, change terms, or quietly stop approving applications. Planning around them means treating the domain as a replaceable pointer, not as permanent infrastructure.\n\nThe 2026 situation, based on what actually happened over the past couple of years:\n\n- **eu.org** still works and is still free, but manual review can take months. It is fine for a project you are just starting; it is not a tool for a site that needs to be live this week.\n- **DigitalPlat's `.us.kg` and `.xx.kg`** paused new registrations after abuse problems. Existing domains kept resolving, but you could not onboard.\n- **`.qzz.io`**, also run under the DigitalPlat umbrella, moved to paid-only.\n- **`dpdns.org`** stayed open for new registrations and became the default choice for people who needed a free subdomain quickly.\n\nThe pattern is obvious: subdomain programs are cheap to launch and expensive to police. Expect any free suffix to have a finite lifespan. The engineering response is to make migration cheap, which is what most of this post is about.\n\n## Why dpdns.org Became the Default\n\n`dpdns.org` is a subdomain service, not a registrar. You get something like `yourname.dpdns.org`, and the provider runs the authoritative nameservers for the zone. You control the records beneath your label.\n\nWhat makes it usable for static hosting:\n\n- New registrations are open, so you can get a name in minutes rather than months.\n- You can delegate to external nameservers if you want, which matters if you already run DNS elsewhere.\n- It works with GitHub Pages, Cloudflare Pages, Netlify, and Vercel without special handling, because those platforms only care about a CNAME pointing at their edge.\n- HTTPS works through the hosting platform's certificate automation, so you do not need to manage TLS for the subdomain yourself.\n\nThe trade-offs are the same as any free subdomain. You do not own the name. The zone owner can change rules. There is no EPP transfer, so you cannot move the domain to another registrar—only the records inside it. And whois is essentially meaningless: the registrant is the provider, and your identity lives only in their control panel.\n\nThat is acceptable for a portfolio, a documentation mirror, or a demo site. It is not acceptable for anything where losing the hostname breaks users, email, or payments.\n\n## The DigitalPlat Registration Flow\n\nThe flow is short, but the DNS step is where people get stuck.\n\n### 1. Register the label\n\nSign in to the DigitalPlat panel, pick the suffix (`dpdns.org`), and enter the label you want. Registration is effectively first-come-first-served, so short labels go fast. Once accepted, the subdomain appears in your dashboard.\n\nAt this point the domain exists but resolves to nothing useful. You have to point it somewhere.\n\n### 2. Choose between provider DNS and external nameservers\n\nTwo paths:\n\n- **Use DigitalPlat's nameservers.** You add A, AAAA, or CNAME records in their panel. Simplest option, and fine for a single static site.\n- **Delegate to external nameservers.** You set NS records for your subdomain to, say, Cloudflare's nameservers, then manage everything there. More control, and it makes the next migration easier because your records live in a zone you can export.\n\nFor static sites, the second path is worth the extra ten minutes. Keeping records in a zone you control means that when `dpdns.org` changes terms, you only need to repoint the NS delegation—your A records, CNAMEs, and TXT records stay put.\n\n### 3. Point the records at your host\n\nFor GitHub Pages, the pattern is a CNAME from your subdomain to `\u003cuser\u003e.github.io`, plus the custom domain set in the repository settings. For Cloudflare Pages or Netlify, it is a CNAME to the project's assigned hostname. Either way, the record is a single CNAME and the platform handles the certificate.\n\nIf you are delegating to external nameservers, you create that CNAME in the external zone, not in the DigitalPlat panel. Doing it in both places causes conflicting answers and confusing propagation behavior.\n\n### 4. Verify resolution before you announce anything\n\nDo not trust the panel's \"active\" status. Query the authoritative servers directly:\n\n

```python\nimport dns.resolver\n\ndef check_subdomain(name: str, record_type: str = \"CNAME\") -\u003e None:\n    \"\"\"Query the authoritative chain for a subdomain and print what it returns.\"\"\"\n    resolver = dns.resolver.Resolver(configure=False)\n    resolver.nameservers = [\"1.1.1.1\", \"8.8.8.8\"]\n    resolver.lifetime = 5.0\n\n    try:\n        answers = resolver.resolve(name, record_type)\n    except dns.resolver.NoAnswer:\n        print(f\"{name} {record_type}: no answer\")\n        return\n    except dns.resolver.NXDOMAIN:\n        print(f\"{name}: NXDOMAIN\")\n        return\n\n    for rdata in answers:\n        print(f\"{name} {record_type} -\u003e {rdata}\")\n\ncheck_subdomain(\"yourname.dpdns.org\", \"CNAME\")\n```

\n\nIf you delegated NS records, also confirm the delegation itself took effect:\n\n

```python\nimport dns.resolver\n\ndef check_delegation(name: str) -\u003e None:\n    \"\"\"Print the NS records that the parent zone actually serves for a subdomain.\"\"\"\n    resolver = dns.resolver.Resolver(configure=False)\n    resolver.nameservers = [\"1.1.1.1\"]\n    try:\n        for ns in resolver.resolve(name, \"NS\"):\n            print(f\"NS: {ns.target}\")\n    except dns.resolver.NoAnswer:\n        print(f\"{name}: no NS records served by parent\")\n\ncheck_delegation(\"yourname.dpdns.org\")\n```

\n\nIf the NS query returns nothing, the delegation has not propagated or was never saved. Fix that before touching anything downstream—a CNAME in a zone nobody is querying will never resolve.\n\n## What You Give Up\n\nBe explicit about the costs, because they shape how you should use these domains.\n\n**Stability risk.** The zone owner can change policy at any time. `.qzz.io` going paid-only is the concrete example; `.us.kg` pausing registrations is another. Your domain is not a contract, it is a courtesy.\n\n**No whois control.** The registrant record belongs to the provider. You cannot prove ownership through whois, which rules out some verification workflows and makes disputes unwinnable.\n\n**No transfer.** There is no EPP code, so you cannot move the name to another registrar. Migration means registering a new name and repointing traffic.\n\n**Certificate and email constraints.** Let's Encrypt issuance works fine for the web hostname, but you should not build email on a free subdomain. SPF, DKIM, and DMARC on a name you do not own is a deliverability problem waiting to happen.\n\n**Search and reputation.** Shared suffixes carry the reputation of everyone else on them. A subdomain on a suffix with heavy abuse gets treated worse by filters and some mail systems.\n\nNone of this is a reason to avoid free domains for hobby work. It is a reason to keep the domain layer thin, documented, and easy to swap.\n\n## Migration Checklist for Near-Zero Downtime\n\nThe goal is to move from `old-free-domain` to `new-free-domain` without a visible gap for visitors. The trick is to run both names in parallel for a while and cut over only when the new one is fully verified.\n\n### 1. Register the new name and set up records first\n\nRegister the new subdomain and configure its records to point at the same origin as the old one. Your host should accept both hostnames—GitHub Pages and Cloudflare Pages both let you attach multiple custom domains to one project, which is what makes this painless.\n\n### 2. Verify the new name end to end\n\nCheck DNS, then check HTTPS and content:\n\n

```python\nimport ssl\nimport urllib.request\n\ndef verify_site(url: str, expected_marker: str) -\u003e bool:\n    \"\"\"Fetch a URL with certificate validation and confirm a marker string is present.\"\"\"\n    ctx = ssl.create_default_context()\n    try:\n        with urllib.request.urlopen(url, context=ctx, timeout=10) as resp:\n            body = resp.read().decode(\"utf-8\", errors=\"replace\")\n            status = resp.status\n    except Exception as exc:\n        print(f\"{url}: FAILED ({exc})\")\n        return False\n\n    ok = status == 200 and expected_marker in body\n    print(f\"{url}: status={status} marker_found={expected_marker in body}\")\n    return ok\n\nverify_site(\"https://newname.dpdns.org\", \"your-site-title\")\n```

\n\nDo not skip this. A CNAME that resolves but serves a certificate error or a 404 from the wrong project is worse than a domain that has not propagated yet.\n\n### 3. Audit the old domain for hardcoded references\n\nAnything that embeds the old hostname needs updating: canonical tags, Open Graph URLs, sitemap entries, RSS feeds, API base URLs in frontend code, and webhook callbacks. A quick scan over a built static site:\n\n

```python\nfrom pathlib import Path\n\nOLD = \"oldname.dpdns.org\"\nNEW = \"newname.dpdns.org\"\n\ndef find_references(root: str) -\u003e None:\n    \"\"\"Scan built output for the old hostname and report file and line.\"\"\"\n    for path in Path(root).rglob(\"*\"):\n        if not path.is_file() or path.suffix not in {\".html\", \".xml\", \".json\", \".js\", \".css\"}:\n            continue\n        try:\n            text = path.read_text(encoding=\"utf-8\", errors=\"ignore\")\n        except OSError:\n            continue\n        for lineno, line in enumerate(text.splitlines(), start=1):\n            if OLD in line:\n                print(f\"{path}:{lineno}: {line.strip()[:120]}\")\n\nfind_references(\"public\")\n```

\n\nFix these in source, rebuild, and redeploy. If you rely on canonical URLs for search indexing, update them before the cutover so the new domain is the one being indexed.\n\n### 4. Keep the old domain alive as a redirect\n\nLeave the old records in place and configure your host to redirect the old hostname to the new one with a 301. This preserves inbound links and search signals. On GitHub Pages you get this by keeping both custom domains attached and adding a redirect at the edge; on Cloudflare you can do it with a redirect rule.\n\n### 5. Cut over traffic sources\n\nUpdate anything you control that points at the old name: bookmarks in docs, links in README files, profile URLs, and any CI jobs that curl the site for smoke tests. This is the slow part, and it is why step 4 matters—the redirect covers the tail.\n\n### 6. Retire the old name later, not immediately\n\nKeep the redirect for at least a few months. Free domains can be reclaimed if the provider changes policy, so do not assume the redirect will live forever. The point is to make the new name fully self-sufficient before the old one disappears.\n\n## Keeping the Next Migration Cheap\n\nTwo habits make all of this much easier:\n\n- **Put records in a zone you control.** Delegate NS from the free subdomain to a DNS provider you can export from. When the suffix changes, you repoint one delegation instead of rebuilding every record.\n- **Never hardcode the hostname.** Use a single config value for the site origin and derive canonical URLs, sitemaps, and feeds from it. The scan script above exists because this rule gets broken; a build-time constant means it only has to be changed once.\n\nFree domains are a reasonable choice for hobby infrastructure as long as you treat them as a pointer you can move. The providers will keep changing; your build process should not care.\n\nMore notes like this ship every week on this site.\n\n---\n\n## Daily Picks\n\n\u003e The following pairs are selected from the multi-timeframe trend scanner (Gate.io futures) and are for technical-analysis study only — not investment advice.\n\u003e Data updated: 2026-10-21 08:46:52\n\n### Short\n\n| Pair | Signal | Price | Take Profit | Stop Loss | R/R |\n|------|--------|-------|-------------|-----------|-----|\n| JCT | 热点跟涨 | $0.0047 | $0.0043 | $0.0048 | 1:1.9 |\n\n_1 picks selected. Scanner runs every 15 minutes._","positive_reactions_count":4,"cover_image":null,"tag_list":["freedomain","euorg","dpdnsorg","digitalplat"],"canonical_url":"https://www.hkcode.dpdns.org/en/posts/23.html","reading_time_minutes":7,"user":{"name":"mattleeee","username":"mattleeee","twitter_username":null,"github_username":"mattleeee","user_id":4160029,"website_url":null,"profile_image":"https://media2.dev.to/dynamic/image/width=640,height=640,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png","profile_image_90":"https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png"}},{"type_of":"article","id":4880524,"title":"Cross-Posting Automation: One Article, Three Platforms","description":"Liquid syntax error: 'raw' tag was never closed","published":true,"published_at":"2026-10-20T01:01:44.680Z","slug":"cross-posting-automation-one-article-three-platforms-42e8","path":"/mattleeee/cross-posting-automation-one-article-three-platforms-42e8","url":"https://dev.to/mattleeee/cross-posting-automation-one-article-three-platforms-42e8","comments_count":0,"public_reactions_count":0,"page_views_count":50,"published_timestamp":"2026-10-20T01:01:44Z","body_markdown":"Publishing the same article to three places sounds trivial until you actually build it. The canonical copy lives on a self-hosted static site built from Markdown and pushed to GitHub Pages. Juejin accepts a bearer token and has an editor-friendly draft API. CSDN doesn't really offer a supported write API, so you drive it with cookies and the same internal endpoints the web editor uses. Three different auth models, three different content rules, three different ways to fail.\n\nThe design principle that makes this manageable: **the canonical copy is the source of truth, and every secondary platform is best-effort.** If Juejin is down, that is a Juejin problem, not an article problem. The run should still succeed.\n\n## The pipeline shape\n\nA publish run has four phases:\n\n1. **Render and verify** — build the site locally, check the Markdown converts cleanly, confirm required front matter exists.\n2. **Publish canonical** — commit and push to GitHub. This is the only step allowed to fail the run.\n3. **Fan out** — attempt each secondary platform independently. Each returns a structured result, never raises.\n4. **Summarize** — push one notification with per-platform status.\n\nThe key word in phase 3 is *independently*. No shared try/except around the loop, no early exit on first failure. Each platform adapter gets its own attempt and its own timeout.\n\n

```python\nfrom dataclasses import dataclass, field\nfrom enum import Enum\n\nclass Status(str, Enum):\n    OK = \"ok\"\n    SKIPPED = \"skipped\"\n    AUTH_EXPIRED = \"auth_expired\"\n    FAILED = \"failed\"\n    TIMEOUT = \"timeout\"\n\n@dataclass\nclass PublishResult:\n    platform: str\n    status: Status\n    url: str | None = None\n    detail: str = \"\"\n\n    @property\n    def needs_attention(self) -\u003e bool:\n        return self.status in (Status.AUTH_EXPIRED, Status.FAILED, Status.TIMEOUT)\n```

\n\nAdapters never raise past their boundary. They translate exceptions into a `Status`. That single rule removes most of the orchestration complexity downstream.\n\n## Canonical publish first, and only it can fail the run\n\nThe static site build is deterministic and cheap to verify. Render the article, check that the output HTML contains the expected title and at least one code block, then push.\n\n

```python\nimport subprocess\nfrom pathlib import Path\n\ndef publish_canonical(article: Path) -\u003e PublishResult:\n    build = subprocess.run(\n        [\"python\", \"-m\", \"build_site\", \"--strict\", str(article)],\n        capture_output=True, text=True, timeout=120,\n    )\n    if build.returncode != 0:\n        return PublishResult(\"github-pages\", Status.FAILED,\n                             detail=build.stderr[-2000:])\n\n    commit = subprocess.run(\n        [\"git\", \"add\", \"-A\"], capture_output=True, text=True,\n    )\n    commit = subprocess.run(\n        [\"git\", \"commit\", \"-m\", f\"publish: {article.stem}\"],\n        capture_output=True, text=True,\n    )\n    if commit.returncode != 0 and \"nothing to commit\" not in commit.stdout:\n        return PublishResult(\"github-pages\", Status.FAILED,\n                             detail=commit.stdout + commit.stderr)\n\n    push = subprocess.run(\n        [\"git\", \"push\"], capture_output=True, text=True, timeout=120,\n    )\n    if push.returncode != 0:\n        return PublishResult(\"github-pages\", Status.FAILED,\n                             detail=push.stderr[-2000:])\n\n    return PublishResult(\"github-pages\", Status.OK,\n                         url=f\"https://example.dev/posts/{article.stem}/\")\n```

\n\nIf this returns `FAILED`, the orchestrator stops and exits non-zero. Nothing gets posted anywhere else, because posting a secondary copy of an article that isn't live on your own site is backwards. Once canonical succeeds, the run is considered successful regardless of what happens next.\n\n## Cookie hygiene for CSDN\n\nThe CSDN adapter needs a session cookie. The tempting shortcut is to paste the cookie string into a CI variable or a shell script. Don't. Cookie strings are long, contain characters that break shell quoting, rotate silently, and end up in logs. Keep them in a JSON file that is gitignored and injected at runtime.\n\n

```json\n{\n  \"cookies\": [\n    {\"name\": \"UserName\", \"value\": \"...\"},\n    {\"name\": \"UserToken\", \"value\": \"...\"},\n    {\"name\": \"SESSION\", \"value\": \"...\"}\n  ],\n  \"saved_at\": \"2025-01-14T09:30:00Z\"\n}\n```

\n\nLoad it with a small helper that validates shape before use:\n\n

```python\nimport json\nfrom pathlib import Path\n\nCOOKIE_FILE = Path.home() / \".config\" / \"crosspost\" / \"csdn.json\"\n\ndef load_cookies(path: Path = COOKIE_FILE) -\u003e list[dict]:\n    if not path.exists():\n        raise FileNotFoundError(f\"cookie file missing: {path}\")\n    data = json.loads(path.read_text(encoding=\"utf-8\"))\n    cookies = data.get(\"cookies\")\n    if not isinstance(cookies, list) or not cookies:\n        raise ValueError(\"cookie file has no usable cookies\")\n    for c in cookies:\n        if not {\"name\", \"value\"} \u003c= c.keys():\n            raise ValueError(f\"malformed cookie entry: {c.get('name')!r}\")\n    return cookies\n```

\n\nTwo rules that save pain later:\n\n- **Never log cookie values.** Log names and count only. When you debug, you need to know *which* cookie is missing, not its value.\n- **Keep the file out of git and out of shell history.** One file, one path, one loader. If you refresh cookies, you overwrite the file, and nothing else changes.\n\n## Detect expiry before you publish\n\nThe worst failure mode is a half-published article: the draft gets created, then the publish call fails because the session expired, leaving a broken draft behind. Check auth *first* with a cheap read call.\n\nFor CSDN, hit an endpoint that requires the session and returns the current user. For Juejin, call the token-validated \"me\" endpoint. Both are cheap and both fail fast.\n\n

```python\nimport httpx\n\ndef check_csdn_session(cookies: list[dict]) -\u003e Status:\n    jar = {c[\"name\"]: c[\"value\"] for c in cookies}\n    try:\n        r = httpx.get(\n            \"https://bizapi.csdn.net/blog-console-api/v1/user/info\",\n            cookies=jar,\n            headers={\"Referer\": \"https://mp.csdn.net/\"},\n            timeout=15,\n        )\n    except httpx.TimeoutException:\n        return Status.TIMEOUT\n\n    if r.status_code in (401, 403):\n        return Status.AUTH_EXPIRED\n    if r.status_code != 200:\n        return Status.FAILED\n    body = r.json()\n    if body.get(\"code\") != 200:\n        return Status.AUTH_EXPIRED\n    return Status.OK\n```

\n\nWhen the check returns `AUTH_EXPIRED`, the adapter returns that status immediately and does not touch the article. The summary then tells you to refresh that one cookie file, and you don't have to clean up a stray draft.\n\n## Content rules differ, so normalize once\n\nJuejin and CSDN both accept Markdown, but not the same Markdown. Differences that bite in practice:\n\n- **Images.** Juejin requires images to be hosted on its own CDN; external URLs render inconsistently. CSDN tolerates external images but strips some attributes. The safe move is to upload images once to each platform and rewrite URLs per target.\n- **Code fences.** Both handle fenced code, but CSDN's editor sometimes mangles nested fences. Flatten any triple-backtick-inside-triple-backtick before sending.\n- **Front matter.** Neither platform wants your YAML front matter. Strip it before the body leaves your pipeline.\n- **Length and title limits.** Both truncate long titles silently. Validate title length before the call, not after.\n\nDo the normalization in one place, keyed by platform, so the adapters stay dumb:\n\n

```python\nimport re\n\nFRONT_MATTER = re.compile(r\"\\A---\\n.*?\\n---\\n\", re.DOTALL)\nNESTED_FENCE = re.compile(r\"```

`.*?`

```\", re.DOTALL)\n\ndef normalize(markdown: str, platform: str) -\u003e str:\n    body = FRONT_MATTER.sub(\"\", markdown, count=1)\n    body = NESTED_FENCE.sub(lambda m: m.group(0).replace(\"```

`\", \"```\"), body)\n    if platform in (\"juejin\", \"csdn\"):\n        body = rewrite_images(body, platform)\n    return body.strip()\n```\n\n`rewrite_images` uploads each referenced image to the platform's image API and swaps the URL. Cache the mapping by content hash so a re-run doesn't re-upload the same image.\n\n## The orchestrator\n\nWith adapters that never raise, the fan-out is a short loop. The one thing worth adding is a per-platform timeout so a hanging CSDN request can't stall the run.\n\n```python\nimport concurrent.futures as cf\n\nADAPTERS = {\n    \"juejin\": publish_juejin,\n    \"csdn\": publish_csdn,\n}\n\ndef fan_out(markdown: str, meta: dict) -\u003e list[PublishResult]:\n    results: list[PublishResult] = []\n    with cf.ThreadPoolExecutor(max_workers=len(ADAPTERS)) as pool:\n        futures = {\n            pool.submit(safe_publish, name, fn, markdown, meta): name\n            for name, fn in ADAPTERS.items()\n        }\n        for fut in cf.as_completed(futures, timeout=180):\n            results.append(fut.result())\n    return results\n\ndef safe_publish(name, fn, markdown, meta) -\u003e PublishResult:\n    try:\n        return fn(markdown, meta)\n    except Exception as exc:\n        return PublishResult(name, Status.FAILED, detail=repr(exc)[:500])\n```\n\n`safe_publish` is the backstop. Adapters should already handle their own errors, but if one slips through, the run still finishes and the summary still reflects reality.\n\n## The summary push\n\nThis is the part people skip, and it's the part that makes the system usable. One notification, per-platform status, links where available.\n\n```python\ndef format_summary(results: list[PublishResult]) -\u003e str:\n    lines = []\n    for r in sorted(results, key=lambda x: (not x.needs_attention, x.platform)):\n        icon = {Status.OK: \"OK\", Status.SKIPPED: \"--\",\n                Status.AUTH_EXPIRED: \"AUTH\", Status.FAILED: \"FAIL\",\n                Status.TIMEOUT: \"TIME\"}[r.status]\n        line = f\"[{icon}] {r.platform}\"\n        if r.url:\n            line += f\" -\u003e {r.url}\"\n        if r.detail:\n            line += f\" ({r.detail[:120]})\"\n        lines.append(line)\n    return \"\\n\".join(lines)\n```\n\nSort attention-needed platforms to the top. If the first line says `[AUTH] csdn`, you know exactly what to do before reading anything else. The notification goes through whatever channel you already use — a webhook, a bot, an email — and the payload is just this string plus the canonical URL.\n\nExit code logic is one line:\n\n```python\ncanonical_ok = canonical.status == Status.OK\nsys.exit(0 if canonical_ok else 1)\n```\n\nSecondary failures never affect the exit code. They affect the summary, which is where you want them.\n\n## What this buys you\n\nThe pipeline now has one hard dependency (your own site) and two soft ones. A Juejin outage costs you nothing. An expired CSDN cookie costs you one line in a notification and one file refresh. The article is never half-published, because auth is checked before any write, and the canonical copy is always live first.\n\nThe pattern generalizes: any time you fan out to external services with unequal reliability, make the reliable step gate the run, make the unreliable steps independent, and put the truth in one summary instead of scattered logs.\n\nMore notes like this ship every week on this site.\n\n---\n\n## Daily Picks\n\n\u003e The following pairs are selected from the multi-timeframe trend scanner (Gate.io futures) and are for technical-analysis study only — not investment advice.\n\u003e Data updated: 2026-10-20 08:46:52\n\n### Long\n\n| Pair | Signal | Price | Take Profit | Stop Loss | R/R |\n|------|--------|-------|-------------|-----------|-----|\n| PUMP | 高质量标的做多 | $0.0044 | $0.0046 | $0.0044 | 1:1.6 |\n\n_1 picks selected. Scanner runs every 15 minutes._","positive_reactions_count":0,"cover_image":null,"tag_list":["crossposting","githubpages","juejin","csdn"],"canonical_url":"https://www.hkcode.dpdns.org/en/posts/22.html","reading_time_minutes":6,"user":{"name":"mattleeee","username":"mattleeee","twitter_username":null,"github_username":"mattleeee","user_id":4160029,"website_url":null,"profile_image":"https://media2.dev.to/dynamic/image/width=640,height=640,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png","profile_image_90":"https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png"}},{"type_of":"article","id":4874631,"title":"Generating Technical Blog Posts with the DeepSeek API","description":"Liquid syntax error: Unknown tag 'endraw'","published":true,"published_at":"2026-10-19T01:01:25.547Z","slug":"generating-technical-blog-posts-with-the-deepseek-api-10l4","path":"/mattleeee/generating-technical-blog-posts-with-the-deepseek-api-10l4","url":"https://dev.to/mattleeee/generating-technical-blog-posts-with-the-deepseek-api-10l4","comments_count":0,"public_reactions_count":0,"page_views_count":50,"published_timestamp":"2026-10-19T01:01:25Z","body_markdown":"## Why the Prompt Is the Whole QA Process\n\nA daily pipeline that publishes one or two articles per day in two languages, with no human in the loop, has an uncomfortable property: the prompt is not a suggestion, it is the specification. If the prompt is vague, the output is vague, and it goes live. There is no editor to catch a missing code block, an H1 that duplicates the page template, or a 400-word stub that technically satisfies \"write an article.\"\n\nThe setup I run is boring on purpose. A Python job assembles a prompt from three parts, calls the DeepSeek API, validates the result, derives an excerpt, and records the publication in a manifest. Everything below is about making each of those steps deterministic enough that \"nobody reviews it\" stops being a risk.\n\n## Anatomy of the Prompt\n\nThe prompt has three layers, and the boundaries between them matter.\n\n### Layer 1: The fixed persona\n\nThe persona is a constant string shared by every article. It sets voice and, more importantly, sets constraints the model can't negotiate away. \"Practical engineer, code-first, no fluff\" is short, but it's doing real work: it biases the model toward concrete examples and away from the \"in today's fast-paced world\" register that LLMs default to.\n\n```python\nPERSONA = (\n    \"You are a professional software engineer writing for a technical blog. \"\n    \"Your style is practical, concise, and code-first. You write real problems \"\n    \"and real fixes, never marketing copy. You never pad with filler sentences.\"\n)\n```\n\nKeep this in a module-level constant. If you edit it weekly, you can't attribute changes in output quality to anything.\n\n### Layer 2: Hard requirements, numbered\n\nNumbered requirements are easier for the model to follow than prose paragraphs, and easier for you to diff when something goes wrong. Each one should be checkable by a script after the fact.\n\n```python\nREQUIREMENTS = \"\"\"\\\n1. Output pure Markdown. No HTML, no JSON wrapper, no preamble.\n2. Do not use a top-level # heading; the page template renders the title.\n3. Use ## and ### headings for structure.\n4. Length: 1200-1800 words.\n5. Include concrete Python code examples in ```python fenced blocks.\n6. Explain methodology only. No disclaimers, no investment advice.\n7. Do not recommend any specific trade, coin, or stock.\n8. Start directly with the subject; no greetings.\n9. Close with one short sentence.\n\"\"\"\n```\n\nRequirement 2 exists because the model will otherwise start with `# {title}` roughly every time, and then your rendered page shows the title twice. Requirement 4 is a range, not a minimum, because \"write at least 1200 words\" invites padding. A range gives the model a ceiling too.\n\n### Layer 3: Topic background\n\nThis is the part that separates usable drafts from generic ones. If you prompt with only a title, the model writes from its prior — which means plausible-sounding, unspecific, and occasionally invented. A short paragraph of real context (\"we run a pipeline that publishes daily in two languages with no reviewer; here's what broke and how we fixed it\") anchors the article in evidence. The model stops inventing scenarios and starts explaining the one you handed it.\n\n```python\ndef build_prompt(title: str, category: str, keywords: list[str],\n                 background: str) -\u003e str:\n    kw = \", \".join(keywords)\n    return f\"\"\"{PERSONA}\n\nWrite one blog post.\n\nTitle: {title}\nCategory: {category}\nKeywords: {kw}\n\nRequirements:\n{REQUIREMENTS}\n\nTopic background (write from this; do not invent facts beyond it):\n{background}\n\"\"\"\n```\n\nThe background paragraph is the highest-leverage input you control. Write two or three sentences of genuine specifics — what the system does, what failed, what the constraint was — and the draft will carry those specifics through. Leave it out and you get an article about \"LLM integration\" in general, which nobody needs.\n\n## Sampling Parameters\n\nTwo knobs matter here: `temperature` and `max_tokens`.\n\nFor this kind of structured, requirement-heavy output, `temperature` around 0.7 is the sweet spot. Lower (`0.2`–`0.4`) makes the model rigid; it follows the format but the prose flattens into a checklist. Higher (`1.0`+) starts drifting on the hard requirements — you'll see the H1 come back, or the word count overshoot. `0.7` keeps some variation between the two daily articles while staying compliant.\n\n`max_tokens` needs headroom. 1200–1800 words is roughly 1600–2400 tokens for English; add the code blocks and you're looking at 3000+ tokens of output. Set `max_tokens` to `4096` and check the finish reason — if it's `length`, the article was truncated mid-sentence and must be retried, not published.\n\n```python\nimport os\nfrom openai import OpenAI\n\nclient = OpenAI(\n    api_key=os.environ[\"DEEPSEEK_API_KEY\"],\n    base_url=\"https://api.deepseek.com\",\n)\n\ndef call_model(prompt: str) -\u003e tuple[str, str]:\n    resp = client.chat.completions.create(\n        model=\"deepseek-chat\",\n        messages=[{\"role\": \"user\", \"content\": prompt}],\n        temperature=0.7,\n        max_tokens=4096,\n    )\n    choice = resp.choices[0]\n    return choice.message.content, choice.finish_reason\n```\n\nThe DeepSeek API is OpenAI-compatible, so the standard client works with a `base_url` override. That also means the retry logic below is portable to any compatible endpoint.\n\n## Retries with Backoff\n\nTimeouts and 5xx responses happen. What you cannot afford is a pipeline that either crashes the whole daily run or silently publishes a half-generated draft. The retry policy needs three properties: bounded attempts, exponential backoff with jitter, and a distinction between retryable and non-retryable failures.\n\n```python\nimport random\nimport time\n\nRETRYABLE = (TimeoutError, ConnectionError)\n\ndef call_with_retry(prompt: str, attempts: int = 4) -\u003e str:\n    last_err = None\n    for i in range(attempts):\n        try:\n            text, finish = call_model(prompt)\n            if finish == \"length\":\n                raise ValueError(\"truncated output\")\n            return text\n        except Exception as e:\n            last_err = e\n            retryable = isinstance(e, RETRYABLE) or \"rate\" in str(e).lower()\n            if not retryable or i == attempts - 1:\n                raise\n            delay = (2 ** i) + random.uniform(0, 0.5)\n            time.sleep(delay)\n    raise RuntimeError(\"unreachable\") from last_err\n```\n\nTwo details worth calling out. First, a truncated output (`finish_reason == \"length\"`) is treated as a failure and retried — publishing a 900-word article that ends mid-sentence is worse than publishing nothing. Second, jitter on the backoff prevents the two language variants from retrying in lockstep and hammering the API at the same instant.\n\nIf you generate both languages concurrently, cap the concurrency (a `ThreadPoolExecutor(max_workers=2)` is fine for two). The rate limit is shared, and a burst of four retries across two languages is how you turn a transient 429 into a sustained one.\n\n## Validating the Draft\n\nRequirements are only real if something checks them. A short validator catches the common failures before they reach the site.\n\n```python\nimport re\n\ndef validate(text: str) -\u003e list[str]:\n    problems = []\n    if re.search(r\"^# \", text, flags=re.MULTILINE):\n        problems.append(\"contains H1 heading\")\n    words = len(re.findall(r\"\\b\\w+\\b\", text))\n    if not 1100 \u003c= words \u003c= 1900:\n        problems.append(f\"word count {words} out of range\")\n    if \"```python\" not in text:\n        problems.append(\"no python code block\")\n    return problems\n```\n\nThe word-count bounds are slightly wider than the prompt's (1100–1900 vs. 1200–1800) to absorb tokenization differences between your counter and the model's. If validation fails, retry once with the problems appended to the prompt; if it fails again, skip the article and log it. A missing article is a recoverable state; a malformed one on the homepage is not.\n\n## Deriving the Excerpt\n\nYou don't want to pay for a second API call to summarize each article, and you don't want raw Markdown in the meta description. Strip it.\n\n```python\ndef make_excerpt(markdown: str, limit: int = 160) -\u003e str:\n    text = re.sub(r\"```.*?```\", \"\", markdown, flags=re.DOTALL)  # drop code\n    text = re.sub(r\"^#{1,6}\\s+.*$\", \"\", text, flags=re.MULTILINE)  # drop headings\n    text = re.sub(r\"!?\\[([^\\]]*)\\]\\([^)]*\\)\", r\"\\1\", text)  # keep link text\n    text = re.sub(r\"[*_`\u003e]\", \"\", text)\n    text = re.sub(r\"\\s+\", \" \", text).strip()\n    if len(text) \u003c= limit:\n        return text\n    cut = text[:limit].rsplit(\" \", 1)[0]\n    return cut.rstrip(\",.;:\") + \"…\"\n```\n\nDropping code blocks and headings first matters: otherwise the excerpt often starts with a function signature or a section title, which reads badly in a search result. Cutting on a word boundary and trimming trailing punctuation avoids the \"…fixes the retry lo…\" look.\n\n## The Manifest\n\nWith no human reviewing, you need a record of what was published, when, and from which prompt. A `manifest.json` keyed by slug gives you idempotency (the job can check whether today's article already exists and skip), reproducibility (you can see which prompt version produced a given post), and a cheap audit trail.\n\n```python\nimport json\nfrom datetime import date\nfrom pathlib import Path\n\nMANIFEST = Path(\"manifest.json\")\n\ndef load_manifest() -\u003e dict:\n    if MANIFEST.exists():\n        return json.loads(MANIFEST.read_text(encoding=\"utf-8\"))\n    return {}\n\ndef record(entry: dict) -\u003e None:\n    data = load_manifest()\n    data[entry[\"slug\"]] = entry\n    MANIFEST.write_text(\n        json.dumps(data, indent=2, ensure_ascii=False), encoding=\"utf-8\"\n    )\n\ndef already_published(slug: str) -\u003e bool:\n    return slug in load_manifest()\n```\n\nEach entry should carry at least: `slug`, `title`, `lang`, `published_at`, `word_count`, `prompt_hash`, and `model`. The `prompt_hash` is a SHA-256 of the assembled prompt — when a batch of articles suddenly reads differently, you can tell whether the prompt changed or the model did.\n\n```python\nimport hashlib\n\ndef prompt_hash(prompt: str) -\u003e str:\n    return hashlib.sha256(prompt.encode(\"utf-8\")).hexdigest()[:12]\n```\n\nWrite the manifest after the article file is safely on disk, not before. If the process dies between the two writes, you'd rather re-generate than have a manifest entry pointing at nothing.\n\n## Putting the Pieces Together\n\nThe daily job is then a straight line: for each (title, language) pair, check the manifest, build the prompt, call with retry, validate, write the file, derive the excerpt, record the entry. The whole thing is maybe 120 lines, and the only part that requires judgment is the topic background — which is exactly where your attention should go, because it's the one input that determines whether the draft is specific or generic.\n\nThe pattern generalizes beyond blogging. Any unattended LLM pipeline that produces structured artifacts — reports, changelogs, documentation stubs — benefits from the same shape: fixed persona, numbered checkable requirements, evidence-based background, bounded retries, a validator, and a manifest. The model is the least predictable component, so everything around it should be the most.\n\nMore notes like this ship every week on this site.\n\n---\n\n## Daily Picks\n\n\u003e The following pairs are selected from the multi-timeframe trend scanner (Gate.io futures) and are for technical-analysis study only — not investment advice.\n\u003e Data updated: 2026-10-19 08:46:24\n\n### Long\n\n| Pair | Signal | Price | Take Profit | Stop Loss | R/R |\n|------|--------|-------|-------------|-----------|-----|\n| SIREN | 短平快做多 | $0.0307 | $0.0316 | $0.0299 | 1:1.2 |\n\n_1 picks selected. Scanner runs every 15 minutes._","positive_reactions_count":0,"cover_image":null,"tag_list":["deepseek","llm","promptengineering","contentpipeline"],"canonical_url":"https://www.hkcode.dpdns.org/en/posts/21.html","reading_time_minutes":7,"user":{"name":"mattleeee","username":"mattleeee","twitter_username":null,"github_username":"mattleeee","user_id":4160029,"website_url":null,"profile_image":"https://media2.dev.to/dynamic/image/width=640,height=640,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png","profile_image_90":"https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png"}},{"type_of":"article","id":4869606,"title":"Publishing a Static Site to GitHub Pages Without git","description":"Why Skip git at All?   Sometimes git isn't an option. Corporate proxies strip the smart HTTP...","published":true,"published_at":"2026-10-18T01:01:33.059Z","slug":"publishing-a-static-site-to-github-pages-without-git-lk0","path":"/mattleeee/publishing-a-static-site-to-github-pages-without-git-lk0","url":"https://dev.to/mattleeee/publishing-a-static-site-to-github-pages-without-git-lk0","comments_count":0,"public_reactions_count":0,"page_views_count":60,"published_timestamp":"2026-10-18T01:01:33Z","body_markdown":"## Why Skip git at All?\n\nSometimes `git` isn't an option. Corporate proxies strip the smart HTTP protocol. A build agent has no credential helper and no working clone. You're inside a sandbox that only exposes HTTPS to `api.github.com`. The repository still needs to receive files, and GitHub Pages still needs to rebuild.\n\nThe Contents API gives you a REST path to the same effect: read a file, write a file, get a commit. It's slower than `git push` and it produces one commit per file, but it works from anywhere you can make an HTTPS request. This post covers the mechanics that make it behave well: SHA comparison to avoid no-op commits, a deterministic directory walk, rate-limit handling, and a verification read after every write.\n\n## The Blob SHA Is Not a Hash of the File\n\nThe first trap is assuming the `sha` field returned by the Contents API is `sha1(file_bytes)`. It isn't. It's the git object ID of the blob, which is the SHA-1 of a header plus the content:\n\n```python\nsha1(b\"blob \" + str(len(content)).encode() + b\"\\x00\" + content)\n```\n\nThat header is why you can compute the remote file's SHA locally without any network call. If your computed value matches the `sha` from `GET /contents/{path}`, the file on GitHub is byte-identical to your local copy and the `PUT` can be skipped entirely.\n\n```python\nimport hashlib\n\ndef git_blob_sha(content: bytes) -\u003e str:\n    header = b\"blob \" + str(len(content)).encode(\"ascii\") + b\"\\x00\"\n    return hashlib.sha1(header + content).hexdigest()\n```\n\nThis one function is the core of an idempotent deploy. Everything else is plumbing around it.\n\n### Why the skip matters\n\nWithout the comparison, every deploy rewrites every file. GitHub treats each `PUT` as a new commit, even when the content is unchanged, so you get:\n\n- A commit log full of \"Update index.html\" entries with zero diff.\n- A Pages build triggered per commit, so a 200-file site fires 200 builds.\n- Rate-limit budget consumed on writes that changed nothing.\n\nWith the comparison, a redeploy of an unchanged site makes zero `PUT` calls. Only files that actually changed produce commits.\n\n## Reading the Current State\n\n`GET /repos/{owner}/{repo}/contents/{path}` returns either a single object (for a file) or a list (for a directory). For a file you want three fields: `sha`, `content`, and `encoding`. The `content` field is base64 with newlines embedded, so strip whitespace before decoding if you need to compare bytes rather than SHAs.\n\n```python\nimport base64\nimport requests\n\nAPI = \"https://api.github.com\"\n\ndef get_file(session, owner, repo, path, ref=\"main\"):\n    url = f\"{API}/repos/{owner}/{repo}/contents/{path}\"\n    r = session.get(url, params={\"ref\": ref})\n    if r.status_code == 404:\n        return None\n    r.raise_for_status()\n    data = r.json()\n    if isinstance(data, list):\n        raise ValueError(f\"{path} is a directory, not a file\")\n    content = base64.b64decode(data[\"content\"].replace(\"\\n\", \"\"))\n    return {\"sha\": data[\"sha\"], \"content\": content}\n```\n\nA `404` means the file doesn't exist at that ref yet, which is the normal case on first deploy. Treat it as `None`, not as an error.\n\n## Walking the Tree Deterministically\n\nDirectory iteration order differs across filesystems, and it differs again between runs on the same machine. That matters because your deploy log, your commit order, and any partial-failure recovery all depend on a stable sequence. Sort explicitly.\n\n```python\nfrom pathlib import Path\n\ndef iter_files(root: Path, ignore=(\".git\", \".github\", \"__pycache__\")):\n    for p in sorted(root.rglob(\"*\")):\n        if not p.is_file():\n            continue\n        rel = p.relative_to(root)\n        if any(part in ignore for part in rel.parts):\n            continue\n        yield rel.as_posix(), p.read_bytes()\n```\n\nSorting on the full relative path string gives you a depth-first order that's stable across platforms. `rglob` returns paths, but you sort after filtering, so the order doesn't depend on `os.scandir` behavior.\n\nTwo details worth pinning down: use `as_posix()` for the API path so Windows separators never leak into URLs, and read bytes rather than text. Line-ending normalization or encoding guesses will change the blob SHA and defeat the comparison.\n\n## The Push\n\nThe `PUT` endpoint takes `message`, `content` (base64), and `sha` — but only when updating. Omitting `sha` on an existing file returns `409 Conflict`, which is GitHub's way of stopping you from clobbering a concurrent write. Including a stale `sha` also fails, so the read and write need to be close together.\n\n```python\ndef push_file(session, owner, repo, path, content, current, message=None):\n    url = f\"{API}/repos/{owner}/{repo}/contents/{path}\"\n    local_sha = git_blob_sha(content)\n    if current is not None and current[\"sha\"] == local_sha:\n        return {\"status\": \"unchanged\", \"sha\": local_sha}\n\n    body = {\n        \"message\": message or f\"deploy: {path}\",\n        \"content\": base64.b64encode(content).decode(\"ascii\"),\n    }\n    if current is not None:\n        body[\"sha\"] = current[\"sha\"]\n\n    r = session.put(url, json=body)\n    r.raise_for_status()\n    return {\"status\": \"pushed\", \"sha\": r.json()[\"content\"][\"sha\"]}\n```\n\nThe response echoes the new blob SHA under `content.sha`. In principle it equals `local_sha`. In practice, don't assume — verify it (next section).\n\n### Base64 encoding\n\n`base64.b64encode` produces standard base64 with `=` padding, which is what the API expects. The API tolerates newlines in the payload but there's no reason to add them. For large files, the entire payload goes in a JSON body, so the practical ceiling is around 1 MB per file for the standard Contents API — above that you need the Git Data API (blobs, trees, commits) instead.\n\n## Verifying the Write\n\nA `200` from `PUT` means GitHub accepted the request. It does not prove the blob on the default branch matches what you sent — webhooks, branch protection rules, or an intervening push can all change the outcome. Re-fetching the SHA is cheap and closes the loop.\n\n```python\ndef deploy_file(session, owner, repo, path, content, ref=\"main\"):\n    current = get_file(session, owner, repo, path, ref=ref)\n    result = push_file(session, owner, repo, path, content, current)\n    if result[\"status\"] == \"unchanged\":\n        return result\n\n    check = get_file(session, owner, repo, path, ref=ref)\n    if check is None or check[\"sha\"] != git_blob_sha(content):\n        raise RuntimeError(\n            f\"verification failed for {path}: \"\n            f\"expected {git_blob_sha(content)}, got {check and check['sha']}\"\n        )\n    return result\n```\n\nThe read-after-write costs one extra request per changed file. On a deploy where most files are unchanged, that's a handful of requests. On a first deploy of a large site, it doubles the request count — worth it, because a silent mismatch means your Pages site is serving stale content and you won't know until someone notices.\n\n## Rate Limiting\n\nAuthenticated requests get 5,000 per hour against the REST API. A 300-file site with verification and a first-time full deploy costs roughly 300 `GET`s (returning 404 for new files), 300 `PUT`s, and 300 verification `GET`s — about 900 requests. Two deploys like that in an hour and you're at a third of your budget. Repeated deploys with the SHA skip cost 300 `GET`s each, since every file is still read.\n\nHandle `403` and `429` explicitly. GitHub returns `403` with `X-RateLimit-Remaining: 0` when exhausted, and `Retry-After` on secondary limits.\n\n```python\nimport time\n\ndef request_with_backoff(session, method, url, **kwargs):\n    for attempt in range(5):\n        r = session.request(method, url, **kwargs)\n        if r.status_code in (403, 429):\n            retry_after = r.headers.get(\"Retry-After\")\n            wait = int(retry_after) if retry_after else 2 ** attempt\n            time.sleep(wait)\n            continue\n        return r\n    return r\n```\n\nWire this into `get_file` and `push_file` by replacing `session.get` / `session.put` with `request_with_backoff`. Cap the retries — a hard-failing deploy should stop, not spin.\n\nIf you're near the limit, the cheapest optimization is a single `GET /repos/{owner}/{repo}/git/trees/{ref}?recursive=1` call to fetch every blob SHA at once, then compare locally and only `GET`/`PUT` the files that differ. That collapses the read phase from N requests to 1.\n\n## The Commit-Per-File Tradeoff\n\nEvery `PUT` creates a commit. There's no way around that through the Contents API alone. The consequences:\n\n- **History noise.** A 50-file change becomes 50 commits. If you care about readable history, batch the message (`deploy: \u003cpath\u003e [3/50]`) or accept it.\n- **Pages rebuilds.** Pages builds on push to the publishing branch. Sequential commits can queue builds or cancel earlier ones. If your site takes 90 seconds to build, a 200-file deploy can take a long time to settle.\n- **Atomicity.** A failure halfway through leaves the repo in a mixed state — some files new, some old. There's no transaction. Design the deploy to be resumable: because the SHA skip is idempotent, re-running the deploy after a failure picks up exactly where it stopped.\n\nIf any of those are unacceptable, the Git Data API is the answer: create blobs, build a tree, create one commit, update the ref. That's a single commit and a single Pages build, at the cost of more code and a wider blast radius if you get the tree wrong. For small sites and infrequent deploys, per-file commits are fine.\n\n## Putting It Together\n\n```python\ndef deploy_site(root: Path, owner: str, repo: str, ref: str = \"main\"):\n    session = requests.Session()\n    session.headers.update({\n        \"Authorization\": f\"Bearer {token}\",\n        \"Accept\": \"application/vnd.github+json\",\n        \"X-GitHub-Api-Version\": \"2022-11-28\",\n    })\n\n    stats = {\"pushed\": 0, \"unchanged\": 0}\n    for path, content in iter_files(root):\n        result = deploy_file(session, owner, repo, path, content, ref=ref)\n        stats[result[\"status\"]] += 1\n        print(f\"{result['status']:9} {path}\")\n    return stats\n```\n\nRun it twice. The second run should report every file as `unchanged` and make zero writes. If it doesn't, your comparison is broken — most often because you're reading text instead of bytes, or because something in the build pipeline is rewriting files with different line endings between runs.\n\nThat idempotency is the whole point. It makes the deploy safe to retry, keeps the commit log honest, and keeps Pages from rebuilding for nothing.\n\nMore notes like this ship every week on this site.\n\n---\n\n## Daily Picks\n\n\u003e The following pairs are selected from the multi-timeframe trend scanner (Gate.io futures) and are for technical-analysis study only — not investment advice.\n\u003e Data updated: 2026-10-18 08:46:32\n\n### Long\n\n| Pair | Signal | Price | Take Profit | Stop Loss | R/R |\n|------|--------|-------|-------------|-----------|-----|\n| AGT | 弱做多 | $0.0843 | $0.0906 | $0.0822 | 1:3.0 |\n| BTW | 短平快做多 | $2.3242 | $2.3939 | $2.2661 | 1:1.2 |\n\n_2 picks selected. Scanner runs every 15 minutes._","positive_reactions_count":0,"cover_image":null,"tag_list":["githubcontentsapi","sha","base64","idempotentpush"],"canonical_url":"https://www.hkcode.dpdns.org/en/posts/20.html","reading_time_minutes":7,"user":{"name":"mattleeee","username":"mattleeee","twitter_username":null,"github_username":"mattleeee","user_id":4160029,"website_url":null,"profile_image":"https://media2.dev.to/dynamic/image/width=640,height=640,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png","profile_image_90":"https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png"}},{"type_of":"article","id":4865077,"title":"Publishing to Juejin Through Its Web API: The mark_content Lesson","description":"Automating content publishing is a common task for developer-advocates and technical writers. Juejin...","published":true,"published_at":"2026-10-17T01:01:34.362Z","slug":"publishing-to-juejin-through-its-web-api-the-markcontent-lesson-38b8","path":"/mattleeee/publishing-to-juejin-through-its-web-api-the-markcontent-lesson-38b8","url":"https://dev.to/mattleeee/publishing-to-juejin-through-its-web-api-the-markcontent-lesson-38b8","comments_count":0,"public_reactions_count":0,"page_views_count":60,"published_timestamp":"2026-10-17T01:01:34Z","body_markdown":"Automating content publishing is a common task for developer-advocates and technical writers. Juejin (掘金) is a popular Chinese developer community, and like many modern web apps, it relies on an internal HTTP API for its editor. That API is undocumented and changes without notice. This post is a post-mortem of a silent regression: drafts saved with empty bodies and a publish call returning `err_no=1010`. The fix turned out to be a single field name, but finding it required a disciplined reverse-engineering loop that is worth writing down.\n\n## The Symptom\n\nThe publisher had been running for months. It did three things:\n\n1. `POST /api/v1/draft/save` with the article body.\n2. `POST /api/v1/draft/publish` to push the draft live.\n3. Log the returned `article_id` and URL.\n\nThen, without any code change on our side, every run produced a draft with an empty body. The publish step returned:\n\n```json\n{\"err_no\": 1010, \"err_msg\": \"草稿内容为空\"}\n```\n\nTranslated: \"draft content is empty.\" The draft existed, the title was correct, but the body was blank. The API had added a requirement: the Markdown source must be sent in a field called `mark_content`, separate from the rendered HTML field the client had been sending.\n\n## Reverse-Engineering the Regression\n\nThe first move with any undocumented API is to stop guessing and capture what the real client does. Open Juejin in a browser, open DevTools, go to the Network tab, and filter for `Fetch/XHR`. Then perform the action manually — write a draft with some text and hit save.\n\nYou will see a request to something like:\n\n```http\nPOST https://api.juejin.cn/content_api/v1/article_draft/save\n```\n\nRight-click the request, choose **Copy as cURL (bash)**, and paste it into a text editor. This gives you the full request: method, URL, headers, cookies, and the JSON body. Do the same for the publish action. Two captures are enough to reconstruct the whole flow.\n\nThe cURL for the save call looks roughly like this (truncated):\n\n```bash\ncurl 'https://api.juejin.cn/content_api/v1/article_draft/save' \\\n  -H 'content-type: application/json' \\\n  -H 'cookie: sessionid=...; uuid=...' \\\n  -H 'user-agent: Mozilla/5.0 ...' \\\n  --data-raw '{\"title\":\"test\",\"mark_content\":\"# hello\",\"brief_content\":\"\",\"category_id\":\"1\",\"tag_ids\":[]}'\n```\n\nThe key observation: the body contains **both** `mark_content` and, in the full capture, a `html_content` or similar rendered field. The old client only sent the rendered field. The new API validates `mark_content` and rejects the request if it is missing or empty.\n\n### Reading the Error Code\n\n`err_no=1010` is not documented. But the mapping is easy to infer once you reproduce it deliberately:\n\n- Save a draft with `mark_content` present and non-empty → publish succeeds.\n- Save a draft with `mark_content` omitted → publish returns `1010`.\n- Save a draft with `mark_content: \"\"` → publish returns `1010`.\n\nThat three-point experiment tells you the field is required and that empty string is treated as absent. It also tells you the failure is server-side validation, not an auth problem — a 401 or `err_no` in the 1000s with a different message would point elsewhere.\n\n## Replaying the Calls from Python\n\nWith the cURL captures in hand, the translation to `requests` is mechanical. The only care needed is header fidelity: `content-type`, `user-agent`, and the cookie header.\n\n```python\nimport json\nimport requests\n\nBASE = \"https://api.juejin.cn\"\nSAVE_URL = f\"{BASE}/content_api/v1/article_draft/save\"\nPUBLISH_URL = f\"{BASE}/content_api/v1/article_draft/publish\"\n\nHEADERS = {\n    \"content-type\": \"application/json\",\n    \"user-agent\": (\n        \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) \"\n        \"AppleWebKit/537.36 (KHTML, like Gecko) \"\n        \"Chrome/122.0.0.0 Safari/537.36\"\n    ),\n    \"origin\": \"https://juejin.cn\",\n    \"referer\": \"https://juejin.cn/\",\n}\n\n\ndef save_draft(session: requests.Session, title: str, markdown: str,\n               category_id: str, tag_ids: list[str]) -\u003e str:\n    payload = {\n        \"title\": title,\n        \"mark_content\": markdown,      # the field that broke everything\n        \"brief_content\": markdown[:100],\n        \"category_id\": category_id,\n        \"tag_ids\": tag_ids,\n    }\n    resp = session.post(SAVE_URL, headers=HEADERS,\n                        data=json.dumps(payload), timeout=30)\n    resp.raise_for_status()\n    body = resp.json()\n    if body.get(\"err_no\") != 0:\n        raise RuntimeError(f\"save failed: {body}\")\n    return body[\"data\"][\"id\"]\n\n\ndef publish_draft(session: requests.Session, draft_id: str) -\u003e str:\n    payload = {\"draft_id\": draft_id, \"column_ids\": []}\n    resp = session.post(PUBLISH_URL, headers=HEADERS,\n                        data=json.dumps(payload), timeout=30)\n    resp.raise_for_status()\n    body = resp.json()\n    if body.get(\"err_no\") != 0:\n        raise RuntimeError(f\"publish failed: {body}\")\n    return body[\"data\"][\"article_id\"]\n```\n\nTwo details matter here. First, `data=json.dumps(payload)` is used instead of `json=payload`. Both work in most cases, but the explicit form makes the serialization visible and avoids surprises if you later need to control separators or encoding. Second, the session object is reused so cookies persist across the save and publish calls — publishing a draft that was just saved in a different session will fail.\n\n## The Cookie Problem in Windows Batch Launchers\n\nThe publisher was launched from a `.bat` file, which is where the next bug lived. The cookie header contains `sessionid=...; uuid=...`. In a batch file, `%` is the escape character for environment variables. If any cookie value contains a `%`, CMD will try to expand it and mangle the string. Worse, the mangling is silent — you get a 401 or a mysterious `err_no` with no hint about why.\n\nThe fix is to stop passing cookies as a command-line string entirely. Store them in a file and read them at runtime.\n\n`cookies.json`:\n\n```json\n{\n  \"sessionid\": \"your-session-id-here\",\n  \"uuid\": \"your-uuid-here\"\n}\n```\n\nLoader:\n\n```python\nimport json\nfrom pathlib import Path\n\ndef load_session(cookie_file: str = \"cookies.json\") -\u003e requests.Session:\n    data = json.loads(Path(cookie_file).read_text(encoding=\"utf-8\"))\n    session = requests.Session()\n    for name, value in data.items():\n        session.cookies.set(name, value, domain=\".juejin.cn\")\n    return session\n```\n\nNow the `.bat` launcher is trivial and cannot mangle anything:\n\n```bat\n@echo off\npython publish.py --draft drafts\\my-post.md\n```\n\nIf you must keep cookies in the batch file for some reason, double every `%` as `%%`. But the file approach is strictly better: it is readable, diffable, and easy to rotate when the session expires. Never commit `cookies.json` — add it to `.gitignore`. The `sessionid` is a full account credential.\n\n## Batch Backfill: Republishing 20 Historical Posts\n\nOnce the single-post path worked, the obvious next step was to repair the archive. Twenty posts had been published with empty bodies over the broken window. Rather than fix them by hand, a backfill script iterated over local Markdown files, matched each to its existing draft or article ID, and re-saved with `mark_content` populated.\n\nThe matching step is the interesting part. If you did not store the article ID at publish time, you cannot reliably match by title alone. The pragmatic approach is to store a small manifest alongside the drafts:\n\n```json\n{\n  \"my-post.md\": {\"draft_id\": \"7123...\", \"article_id\": \"7124...\"}\n}\n```\n\nThe backfill script then reads the manifest, loads each Markdown file, and re-saves:\n\n```python\nimport json\nfrom pathlib import Path\n\ndef backfill(session, drafts_dir: str, manifest_path: str,\n             category_id: str, tag_ids: list[str]) -\u003e None:\n    manifest = json.loads(Path(manifest_path).read_text(encoding=\"utf-8\"))\n    ok, failed = 0, []\n\n    for filename, entry in manifest.items():\n        path = Path(drafts_dir) / filename\n        if not path.exists():\n            failed.append((filename, \"missing file\"))\n            continue\n\n        markdown = path.read_text(encoding=\"utf-8\")\n        title = markdown.splitlines()[0].lstrip(\"# \").strip()\n\n        try:\n            new_id = save_draft(session, title, markdown,\n                                category_id, tag_ids)\n            entry[\"draft_id\"] = new_id\n            ok += 1\n        except Exception as exc:  # noqa: BLE001\n            failed.append((filename, str(exc)))\n\n    Path(manifest_path).write_text(\n        json.dumps(manifest, ensure_ascii=False, indent=2),\n        encoding=\"utf-8\",\n    )\n    print(f\"backfilled {ok}, failed {len(failed)}\")\n    for name, reason in failed:\n        print(f\"  {name}: {reason}\")\n```\n\nTwo operational notes from running this against 20 posts:\n\n- **Rate-limit yourself.** Insert a `time.sleep(2)` between iterations. Twenty rapid-fire saves can trip a soft throttle, and the failure mode is an ambiguous error rather than a clean 429.\n- **Write the manifest after every success, not at the end.** If the script dies at post 14, you do not want to re-save the first 13. The version above writes once at the end for brevity; in production, flush after each item.\n\nThe run republished all 20 posts in a single pass. The only failures were two files whose titles had been edited in the CMS after the original publish, so the manifest IDs pointed at stale drafts. Those were resolved by re-creating the drafts and updating the manifest.\n\n## Lessons for Any Undocumented API\n\nThe regression was a one-field change, but the process that found it generalizes:\n\n1. **Capture before you theorize.** Copy-as-cURL from DevTools is the ground truth. The browser is the only client the server is guaranteed to support.\n2. **Diff the capture against your payload.** The missing field was visible the moment the two were placed side by side.\n3. **Reproduce the error deliberately.** Vary one field at a time until you can trigger and clear the failure on demand. That is what turns `err_no=1010` from a mystery into a known precondition.\n4. **Keep credentials out of shell strings.** Files beat command lines, especially on Windows.\n5. **Make batch operations idempotent and resumable.** Persist progress as you go.\n\nThe publisher has run clean since. The `mark_content` field is now set in one place, and the backfill script lives in the repo as a reusable repair tool for the next time the API quietly grows a new requirement.\n\nMore notes like this ship every week on this site.","positive_reactions_count":0,"cover_image":null,"tag_list":["juejin","undocumentedapi","markcontent","publishing"],"canonical_url":"https://www.hkcode.dpdns.org/en/posts/19.html","reading_time_minutes":6,"user":{"name":"mattleeee","username":"mattleeee","twitter_username":null,"github_username":"mattleeee","user_id":4160029,"website_url":null,"profile_image":"https://media2.dev.to/dynamic/image/width=640,height=640,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png","profile_image_90":"https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png"}},{"type_of":"article","id":4859141,"title":"Making .bat Files Bulletproof Across Windows Machines","description":"Batch files are the glue of Windows automation and the most fragile part of it. They look trivial: a...","published":true,"published_at":"2026-10-16T01:01:28.207Z","slug":"making-bat-files-bulletproof-across-windows-machines-4oo8","path":"/mattleeee/making-bat-files-bulletproof-across-windows-machines-4oo8","url":"https://dev.to/mattleeee/making-bat-files-bulletproof-across-windows-machines-4oo8","comments_count":0,"public_reactions_count":0,"page_views_count":90,"published_timestamp":"2026-10-16T01:01:28Z","body_markdown":"Batch files are the glue of Windows automation and the most fragile part of it. They look trivial: a text file with a few commands, saved as `.bat`, double-clicked. Then you copy that file to another machine and it explodes — sometimes literally, with a console full of `'xxx' is not recognized as an internal or external command`. The failures are rarely about the commands. They are about bytes: the encoding, the line endings, the codepage, and the invisible characters that a text editor added without asking.\n\nThis post covers the hardening rules I now apply to every `.bat` I ship, why each rule exists, and how each one has broken a real deployment. At the end there is a byte-level verification script you can run before shipping.\n\n## The failure mode is always the same\n\nA batch file is not parsed by a compiler. `cmd.exe` reads it line by line, byte by byte, and interprets what it finds. There is no schema, no lint, no type system. If a byte is unexpected, `cmd` either ignores it, misinterprets it, or feeds it to a command that does something you did not intend.\n\nThat means every rule below is really the same rule: control the bytes.\n\n## Rule 1: Pure ASCII content\n\nThe single most common cause of \"works on my machine\" batch failures is non-ASCII text. A Chinese comment, a curly quote pasted from a document, an em dash — all of it becomes a liability the moment the file leaves the editor that created it.\n\nHere is the mechanism. `cmd.exe` reads the batch file using the console's current codepage, not a fixed encoding declared in the file. On a Chinese Windows install that is typically 936 (GBK). On a US install it is 437 or 1252. On a modern developer machine it might be 65001 (UTF-8) if someone ran `chcp 65001` earlier in the session. The same bytes are decoded differently depending on the machine.\n\nA UTF-8 Chinese comment like `:: 检查目录` becomes three bytes per character. Under codepage 936, those bytes decode into different characters, and — worse — some byte sequences contain `0x5C` (`\\`) or `0x7C` (`|`), which `cmd` treats as syntax. I have seen a comment line turn into a pipe and split a command in half. The error message pointed at a line that looked completely innocent.\n\nThe fix is boring: keep the `.bat` pure ASCII. Comments in English, no accented characters, no smart quotes, no emoji. If you genuinely need localized documentation, put it in a companion `.txt` file next to the script:\n\n```batchfile\ndeploy.bat\ndeploy.notes.zh-CN.txt\n```\n\nThe `.txt` is never executed, so its encoding does not matter to `cmd`. The `.bat` stays safe. This also has a nice side effect: the script becomes readable to anyone on the team regardless of their Windows language settings.\n\nIf you must print localized text from the batch file itself, see Rule 3 — but keep the *file content* ASCII even then, and emit the localized strings via a separate mechanism.\n\n### Checking for non-ASCII bytes\n\nThis is the check I run most often, because it catches the problem before it ever reaches a machine:\n\n```python\nfrom pathlib import Path\n\ndef find_non_ascii(path: Path) -\u003e list[tuple[int, int, int]]:\n    \"\"\"Return (line_no, col_no, byte) for every non-ASCII byte.\"\"\"\n    hits = []\n    data = path.read_bytes()\n    line, col = 1, 0\n    for b in data:\n        if b == 0x0A:\n            line += 1\n            col = 0\n            continue\n        col += 1\n        if b \u003e 0x7F:\n            hits.append((line, col, b))\n    return hits\n\nfor line, col, byte in find_non_ascii(Path(\"deploy.bat\")):\n    print(f\"line {line}, col {col}: 0x{byte:02X}\")\n```\n\nEven one hit is enough to reject the file. There is no upside to a non-ASCII byte in a `.bat`.\n\n## Rule 2: CRLF line endings, always\n\nUnix line endings (`LF`, `0x0A`) are the second classic killer. `cmd.exe` expects `CRLF` (`0x0D 0x0A`). What happens with bare `LF` depends on the command, which is what makes it so nasty.\n\nSimple commands often work. But `goto` labels, `if` blocks with parentheses, and `for` loops can behave unpredictably, because `cmd` reads ahead and the missing carriage return shifts its notion of where a line ends. The symptom is usually a `goto` that jumps to nowhere, or a block that executes the wrong branch. I once spent an afternoon on a `for /f` loop that silently skipped its body on one machine and worked on another — the only difference was that the broken file had been checked out with `core.autocrlf=false`.\n\nThe root cause is usually Git. If `.bat` files are committed with `LF` and checked out on Linux or macOS, they arrive on Windows with `LF`. The fix is a `.gitattributes` entry:\n\n```ini\n*.bat text eol=crlf\n*.cmd text eol=crlf\n```\n\nThat single line prevents an entire class of incidents. Add it before you have batch files in the repo, not after.\n\n### Verifying line endings in Python\n\n```python\nfrom pathlib import Path\n\ndef check_line_endings(path: Path) -\u003e list[int]:\n    \"\"\"Return line numbers that end with bare LF (no preceding CR).\"\"\"\n    bad = []\n    data = path.read_bytes()\n    line = 1\n    i = 0\n    while i \u003c len(data):\n        b = data[i]\n        if b == 0x0A:\n            if i == 0 or data[i - 1] != 0x0D:\n                bad.append(line)\n            line += 1\n        i += 1\n    return bad\n```\n\nNote that this flags `LF` at the very start of the file too, which is correct — a batch file should not begin with a bare newline.\n\n## Rule 3: Do not rely on the console codepage\n\n`chcp` changes the active console codepage, and it is tempting to write `chcp 65001` at the top of a script so that UTF-8 output works. This is a trap for two reasons.\n\nFirst, `chcp 65001` historically broke more than it fixed on older Windows versions, and behavior still varies by build. Second, and more importantly, the codepage at the time `cmd` *reads the file* is what matters for parsing — changing it inside the script only affects output after that point. If the file itself contains non-ASCII bytes, you have already lost before `chcp` runs.\n\nThe working pattern is:\n\n1. Keep the `.bat` pure ASCII (Rule 1).\n2. If you must print localized text, call `chcp` as the very first command, and only when needed.\n3. Better: do not print localized text from the batch file at all. Let a Python helper do it.\n\n```bat\n@echo off\nsetlocal\nrem Pure ASCII only. No chcp unless localized output is required.\npython \"%~dp0report.py\" %*\nif errorlevel 1 (\n    echo report.py failed with code %errorlevel%\n    exit /b %errorlevel%\n)\nendlocal\n```\n\nThe Python helper handles its own encoding, which is far more controllable:\n\n```python\nimport sys\n\ndef safe_print(text: str) -\u003e None:\n    \"\"\"Print to stdout, degrading gracefully if the console can't encode it.\"\"\"\n    try:\n        print(text)\n    except UnicodeEncodeError:\n        enc = sys.stdout.encoding or \"ascii\"\n        print(text.encode(enc, errors=\"replace\").decode(enc))\n```\n\nThis keeps the batch file dumb and portable, and puts the encoding decisions where they can actually be tested.\n\n## Rule 4: Quote every path, and derive paths from `%USERPROFILE%`\n\nUnquoted paths break on spaces. This is old news, but the subtler problem is *which* paths you use. Hardcoding `C:\\Users\\Alice` works until the script runs as a service account, on a machine where the profile is on `D:`, or under a different user. Derive everything from environment variables:\n\n```bat\n@echo off\nsetlocal\nset \"APPDIR=%USERPROFILE%\\AppData\\Local\\MyTool\"\nset \"LOGDIR=%APPDIR%\\logs\"\nif not exist \"%APPDIR%\" mkdir \"%APPDIR%\"\nif not exist \"%LOGDIR%\" mkdir \"%LOGDIR%\"\n```\n\nEvery expansion is quoted. `%USERPROFILE%` can contain spaces (rare but legal), and `%APPDIR%` certainly can. An unquoted `mkdir %APPDIR%` with a space in the path creates two directories in the current folder and then fails on the second one.\n\n## Rule 5: Use `%~dp0` for self-locating paths\n\nScripts that reference sibling files — a `.py` helper, a config, a log — should locate them relative to the script, not relative to the current working directory. `%~dp0` expands to the drive and path of the batch file itself, including the trailing backslash.\n\n```bat\n@echo off\nsetlocal\npython \"%~dp0report.py\" --config \"%~dp0config.json\"\n```\n\nWithout this, double-clicking the `.bat` from Explorer works (cwd is the script folder), but running it from a shortcut or a scheduler task with a different working directory fails. I have seen a scheduled task run at 3 AM with cwd `C:\\Windows\\System32` and fail to find its own config. `%~dp0` fixes it permanently.\n\nOne caveat: `%~dp0` already ends with a backslash, so `\"%~dp0\\file\"` produces a double backslash. It usually works, but write `\"%~dp0file\"` instead.\n\n## Rule 6: No trailing spaces after `set \"VAR=value\"`\n\nThis is the one that costs people hours because the output looks correct. Consider:\n\n```bat\nset \"MODE=prod\" \necho Mode is [%MODE%]\n```\n\nThere is a single trailing space after `prod` on the first line. `cmd` includes it in the value. The variable is now `prod `, not `prod`. Every comparison against `prod` fails, every path built from it has a trailing space, and the `echo` shows `[prod ]` — which is easy to miss when you skim the console.\n\nThe quoted form `set \"VAR=value\"` protects against trailing spaces *before* the assignment and against special characters in the value, but it does not protect against a space *inside* the quotes. There is no way to see it in a normal editor. This is exactly the kind of thing a byte-level check should catch.\n\n```python\nfrom pathlib import Path\nimport re\n\nSET_RE = re.compile(rb'^\\s*set\\s+\"([^=]+)=([^\"]*)\"\\s*$', re.IGNORECASE)\n\ndef find_trailing_space_in_set(path: Path) -\u003e list[tuple[int, bytes, bytes]]:\n    \"\"\"Flag `set \"VAR=value \"` where value ends with a space.\"\"\"\n    hits = []\n    for i, line in enumerate(path.read_bytes().splitlines(), start=1):\n        m = SET_RE.match(line)\n        if m and m.group(2).endswith(b\" \"):\n            hits.append((i, m.group(1), m.group(2)))\n    return hits\n```\n\nAny hit is a bug. Fix the source line; do not try to compensate downstream.\n\n## The pre-ship verification script\n\nAll of the above collapses into one script. Run it in CI or as a pre-commit hook. It reads the file as bytes, which is the only honest way to inspect a batch file.\n\n```python\nfrom pathlib import Path\nimport re\nimport sys\n\nSET_RE = re.compile(rb'^\\s*set\\s+\"([^=]+)=([^\"]*)\"\\s*$', re.IGNORECASE)\n\ndef audit_bat(path: Path) -\u003e list[str]:\n    problems = []\n    data = path.read_bytes()\n\n    # 1. Non-ASCII bytes\n    line, col = 1, 0\n    for b in data:\n        if b == 0x0A:\n            line += 1\n            col = 0\n            continue\n        col += 1\n        if b \u003e 0x7F:\n            problems.append(f\"line {line}: non-ASCII byte 0x{b:02X}\")\n\n    # 2. Bare LF line endings\n    line = 1\n    for i, b in enumerate(data):\n        if b == 0x0A:\n            if i == 0 or data[i - 1] != 0x0D:\n                problems.append(f\"line {line}: bare LF (expected CRLF)\")\n            line += 1\n\n    # 3. Trailing space inside set \"VAR=value \"\n    for i, raw in enumerate(data.splitlines(), start=1):\n        m = SET_RE.match(raw)\n        if m and m.group(2).endswith(b\" \"):\n            problems.append(f\"line {i}: trailing space in set \\\"{m.group(1).decode()}=...\\\"\")\n\n    # 4. Unquoted %USERPROFILE% expansions (heuristic)\n    for i, raw in enumerate(data.splitlines(), start=1):\n        stripped = raw.strip()\n        if stripped.startswith(b\"rem\") or stripped.startswith(b\"::\"):\n            continue\n        for var in (b\"%USERPROFILE%\", b\"%APPDIR%\"):\n            if var in raw and b'\"' not in raw:\n                problems.append(\n                    f\"line {i}: {var.decode()} used without quotes\"\n                )\n\n    return problems\n\nif __name__ == \"__main__\":\n    failed = False\n    for p in sys.argv[1:]:\n        path = Path(p)\n        issues = audit_bat(path)\n        if issues:\n            failed = True\n            print(f\"{path}:\")\n            for issue in issues:\n                print(f\"  {issue}\")\n    sys.exit(1 if failed else 0)\n```\n\nThe heuristic in check 4 is deliberately crude — it flags lines that mention those variables without any quotes at all. It is not a parser, but it catches the mistakes that actually happen. Extend the variable list as your scripts grow.\n\nRun it like this:\n\n```shell\npython audit_bat.py deploy.bat install.bat\n```\n\nExit code 1 means do not ship. Wire it into CI and the class of failures described here mostly disappears.\n\n## What this buys you\n\nNone of these rules are clever. They are the residue of real incidents: a comment that split a command, a `goto` that missed its label, a scheduled task that could not find its config, a variable with an invisible trailing space. Each one is cheap to prevent at the byte level and expensive to debug at runtime.\n\nThe habit that matters is reading batch files as bytes, not as text. Once you do that, the fixes are obvious. More notes like this ship every week on this site.\n\n---\n\n## Daily Picks\n\n\u003e The following pairs are selected from the multi-timeframe trend scanner (Gate.io futures) and are for technical-analysis study only — not investment advice.\n\u003e Data updated: 2026-10-16 08:46:41\n\n### Long\n\n| Pair | Signal | Price | Take Profit | Stop Loss | R/R |\n|------|--------|-------|-------------|-----------|-----|\n| ZEREBRO | 高质量标的做多 | $0.051 | $0.0541 | $0.0494 | 1:2.0 |\n| ARC | 短平快做多 | $0.0359 | $0.037 | $0.035 | 1:1.2 |\n\n_2 picks selected. Scanner runs every 15 minutes._","positive_reactions_count":0,"cover_image":null,"tag_list":["batchfile","ascii","crlf","chcp"],"canonical_url":"https://www.hkcode.dpdns.org/en/posts/18.html","reading_time_minutes":8,"user":{"name":"mattleeee","username":"mattleeee","twitter_username":null,"github_username":"mattleeee","user_id":4160029,"website_url":null,"profile_image":"https://media2.dev.to/dynamic/image/width=640,height=640,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png","profile_image_90":"https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png"}},{"type_of":"article","id":4847169,"title":"PowerShell on Legacy Chinese Windows: UTF-8 with BOM or Bust","description":"Windows PowerShell 5.1 does not guess encodings. When it loads a .ps1 file, it looks for a byte-order...","published":true,"published_at":"2026-10-14T01:01:34.558Z","slug":"powershell-on-legacy-chinese-windows-utf-8-with-bom-or-bust-4g5b","path":"/mattleeee/powershell-on-legacy-chinese-windows-utf-8-with-bom-or-bust-4g5b","url":"https://dev.to/mattleeee/powershell-on-legacy-chinese-windows-utf-8-with-bom-or-bust-4g5b","comments_count":0,"public_reactions_count":0,"page_views_count":140,"published_timestamp":"2026-10-14T01:01:34Z","body_markdown":"Windows PowerShell 5.1 does not guess encodings. When it loads a `.ps1` file, it looks for a byte-order mark. If it finds one, it decodes accordingly. If it does not, it falls back to the system's ANSI code page — and on a Chinese Windows install, that code page is GBK (CP936). Every non-ASCII character in your script is then decoded through the wrong table, and you get mojibake that does not throw an error. It just quietly changes what your script means.\n\nThis is the story of how a path check turned into a week of debugging, and the one rule that fixed it: every `.ps1` file ships as UTF-8 with BOM and CRLF endings.\n\n## The symptom\n\nA deployment script had a line like this:\n\n```powershell\n$configDir = Join-Path $env:ProgramData \"我的工具\\配置\"\nif (-not (Test-Path $configDir)) {\n    New-Item -ItemType Directory -Path $configDir -Force | Out-Null\n}\n```\n\nOn the developer's machine it worked. On the build server it created a directory named something like `我的工具` — or sometimes it created nothing at all, and `Test-Path` returned `$false` for a path that clearly existed when inspected manually in Explorer.\n\nThe reason is straightforward once you see the bytes. The file was saved by an editor as UTF-8 without BOM. PowerShell 5.1 read it as GBK. The UTF-8 bytes for `我` are `E6 88 91`. Decoded as GBK, those three bytes are a valid two-byte character plus a stray byte, producing a different string entirely. PowerShell then passed that corrupted string to the Windows API, which dutifully created a directory with a nonsense name. `Test-Path` on the *intended* name failed because the intended name was never used.\n\nNo exception. No warning. Just a different program than the one you wrote.\n\n## Detecting it: hex-dump the first bytes\n\nThe fastest way to confirm the diagnosis is to look at the raw bytes at the start of the file.\n\n```powershell\n# Show the first 8 bytes of a script file\n$bytes = [System.IO.File]::ReadAllBytes(\".\\deploy.ps1\")[0..7]\n($bytes | ForEach-Object { $_.ToString(\"X2\") }) -join ' '\n```\n\nInterpretation:\n\n- `EF BB BF` — UTF-8 with BOM. PowerShell 5.1 decodes correctly.\n- `FF FE` — UTF-16 LE with BOM. Also fine.\n- `FE FF` — UTF-16 BE with BOM. Fine, rare.\n- Anything else — no BOM. PowerShell 5.1 falls back to the ANSI code page. On a GBK system that means GBK.\n\nIf you see `E6 88 91` at the start of a file containing `我`, you have a BOM-less UTF-8 file, and 5.1 is about to misread it.\n\nA quick Python check across a whole directory is often easier than doing this by hand:\n\n```python\nfrom pathlib import Path\n\nBOMS = {\n    b\"\\xef\\xbb\\xbf\": \"utf-8-sig\",\n    b\"\\xff\\xfe\": \"utf-16-le\",\n    b\"\\xfe\\xff\": \"utf-16-be\",\n}\n\ndef sniff(path: Path) -\u003e str:\n    head = path.read_bytes()[:4]\n    for bom, enc in BOMS.items():\n        if head.startswith(bom):\n            return enc\n    # No BOM. Try strict UTF-8; if it fails, it is almost certainly ANSI/GBK.\n    try:\n        path.read_bytes().decode(\"utf-8\")\n        return \"utf-8 (NO BOM) \u003c-- 5.1 will misread this\"\n    except UnicodeDecodeError:\n        return \"ansi/gbk (NO BOM)\"\n\nfor ps1 in Path(\"scripts\").rglob(\"*.ps1\"):\n    print(f\"{ps1}: {sniff(ps1)}\")\n```\n\nRun this on any legacy repo and the offending files surface immediately. On a Chinese Windows box, the mix of \"utf-8 (NO BOM)\" and \"ansi/gbk\" files is the whole problem in one listing.\n\n## Why BOM-less UTF-8 is a trap on Chinese Windows\n\nThe default encoding for reading files in PowerShell 5.1 is not UTF-8. It is `[System.Text.Encoding]::Default`, which maps to the active ANSI code page. On an English system that is CP1252. On a Simplified Chinese system it is CP936 (GBK). On a Japanese system, CP932. The behavior of the same script file therefore depends on the locale of the machine running it.\n\nThis is the part that makes the bug so expensive. The script is not wrong on every machine — it is wrong on machines whose code page differs from the one the author implicitly assumed. A BOM removes the ambiguity: it tells the parser \"this is UTF-8\" regardless of locale.\n\nThe same trap applies to `Get-Content` and `Set-Content` in 5.1. `Get-Content` without `-Encoding` uses the ANSI default; `Set-Content` without `-Encoding` writes ANSI. Round-tripping a UTF-8 file through those cmdlets silently converts it to GBK.\n\n## Bulk-converting a repo to UTF-8 with BOM\n\nYou can fix a directory of scripts with PowerShell itself. The trick is to read the bytes, decode them with the correct source encoding, and write them back with a BOM.\n\n```powershell\n# convert-to-utf8bom.ps1\nparam(\n    [Parameter(Mandatory)][string]$Root,\n    [string]$SourceEncoding = \"utf-8\"   # use \"gb2312\" for legacy GBK files\n)\n\n$utf8Bom = New-Object System.Text.UTF8Encoding($true)\n\nGet-ChildItem -Path $Root -Recurse -Filter *.ps1 | ForEach-Object {\n    $path  = $_.FullName\n    $bytes = [System.IO.File]::ReadAllBytes($path)\n\n    # Already has a UTF-8 BOM? Skip.\n    if ($bytes.Length -ge 3 -and $bytes[0] -eq 0xEF -and $bytes[1] -eq 0xBB -and $bytes[2] -eq 0xBF) {\n        Write-Host \"skip (already BOM): $path\"\n        return\n    }\n\n    $text = [System.Text.Encoding]::GetEncoding($SourceEncoding).GetString($bytes)\n\n    # Normalize line endings to CRLF.\n    $text = $text -replace \"`r`n\", \"`n\" -replace \"`n\", \"`r`n\"\n\n    [System.IO.File]::WriteAllText($path, $text, $utf8Bom)\n    Write-Host \"converted: $path\"\n}\n```\n\nTwo things matter here. First, `WriteAllText` with a `UTF8Encoding($true)` instance emits the BOM; `UTF8Encoding($false)` does not. Second, you must know the source encoding. If a file is genuinely GBK, decoding it as UTF-8 will throw or produce replacement characters. The Python sniffer above tells you which is which before you run the conversion.\n\nThe same job in Python, which is often more convenient in CI:\n\n```python\nimport sys\nfrom pathlib import Path\n\ndef to_utf8_bom(path: Path, source_encoding: str) -\u003e None:\n    raw = path.read_bytes()\n    if raw.startswith(b\"\\xef\\xbb\\xbf\"):\n        return\n    text = raw.decode(source_encoding)\n    text = text.replace(\"\\r\\n\", \"\\n\").replace(\"\\n\", \"\\r\\n\")\n    path.write_bytes(b\"\\xef\\xbb\\xbf\" + text.encode(\"utf-8\"))\n\nfor p in Path(sys.argv[1]).rglob(\"*.ps1\"):\n    try:\n        to_utf8_bom(p, \"utf-8\")\n    except UnicodeDecodeError:\n        to_utf8_bom(p, \"gbk\")\n    print(f\"converted {p}\")\n```\n\nThe fallback to `gbk` on `UnicodeDecodeError` handles mixed repos where some files were saved as ANSI by an older editor. Once every file is UTF-8 with BOM, the ambiguity is gone.\n\n## PowerShell 7 behaves differently\n\nPowerShell 7 (Core) changed the defaults. `Get-Content` and the script loader default to UTF-8 without requiring a BOM, and `Set-Content` writes UTF-8 without BOM by default. So the same BOM-less file that 5.1 mangles will load correctly in 7.\n\nThat is a feature, and also a trap. If your scripts run under both 5.1 and 7 — which is common on servers that still have Windows PowerShell installed alongside Core — you need the file to work under the stricter reader. UTF-8 with BOM works in both. BOM-less UTF-8 works only in 7. So the rule is not \"use UTF-8\"; the rule is \"use UTF-8 with BOM\" if any 5.1 host might touch the file.\n\nNote also that PowerShell 7 still honors the BOM when present, so adding it does not break anything under Core.\n\n## A pre-flight check before you ship\n\nThe fix is cheap; the discipline is what fails. Put a validator in your build or pre-commit step so a BOM-less file never reaches a 5.1 host.\n\n```python\n#!/usr/bin/env python3\n\"\"\"Fail the build if any .ps1 is not UTF-8 with BOM and CRLF.\"\"\"\nimport sys\nfrom pathlib import Path\n\ndef check(path: Path) -\u003e list[str]:\n    errors = []\n    raw = path.read_bytes()\n\n    if not raw.startswith(b\"\\xef\\xbb\\xbf\"):\n        errors.append(\"missing UTF-8 BOM\")\n    else:\n        try:\n            raw[3:].decode(\"utf-8\")\n        except UnicodeDecodeError as exc:\n            errors.append(f\"not valid UTF-8: {exc}\")\n\n    # CRLF check: reject lone LF outside of the final byte.\n    body = raw[3:] if raw.startswith(b\"\\xef\\xbb\\xbf\") else raw\n    if b\"\\n\" in body.replace(b\"\\r\\n\", b\"\"):\n        errors.append(\"contains LF-only line endings; expected CRLF\")\n\n    return errors\n\nfailed = False\nfor p in sorted(Path(\".\").rglob(\"*.ps1\")):\n    errs = check(p)\n    if errs:\n        failed = True\n        for e in errs:\n            print(f\"{p}: {e}\")\n\nsys.exit(1 if failed else 0)\n```\n\nDrop this into CI as a gate. The failure message is specific enough that the fix is obvious: save the file as UTF-8 with BOM, convert line endings to CRLF, commit.\n\nIf you want the validator to auto-fix instead of fail, reuse the conversion functions above and re-run the check. In a pre-commit hook, auto-fixing is usually the better default — it removes the human from the loop entirely.\n\n## The rule\n\nEvery `.ps1` file that might ever run under Windows PowerShell 5.1 must be saved as UTF-8 with BOM and CRLF line endings. BOM-less UTF-8 is a valid encoding and a broken script on a GBK system. UTF-16 also works but is unusual in a repo and confuses diffs and tooling. Stick to UTF-8 with BOM.\n\nThe failure mode — silent mojibake, locale-dependent behavior, no exception — is what makes this worth a hard rule rather than a convention. A linter catches it in a second; a human catches it after a week of `Test-Path` returning `$false` on a path that exists.\n\nMore notes like this ship every week on this site.","positive_reactions_count":0,"cover_image":null,"tag_list":["powershell","bom","utf8","gbk"],"canonical_url":"https://www.hkcode.dpdns.org/en/posts/17.html","reading_time_minutes":6,"user":{"name":"mattleeee","username":"mattleeee","twitter_username":null,"github_username":"mattleeee","user_id":4160029,"website_url":null,"profile_image":"https://media2.dev.to/dynamic/image/width=640,height=640,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png","profile_image_90":"https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png"}},{"type_of":"article","id":4840826,"title":"UTF-8 vs GBK: The Root Cause of Cross-Machine Script Failures","description":"A script that runs fine on your laptop and dies on the server is almost never a logic bug. It is an...","published":true,"published_at":"2026-10-13T01:01:26.141Z","slug":"utf-8-vs-gbk-the-root-cause-of-cross-machine-script-failures-4l90","path":"/mattleeee/utf-8-vs-gbk-the-root-cause-of-cross-machine-script-failures-4l90","url":"https://dev.to/mattleeee/utf-8-vs-gbk-the-root-cause-of-cross-machine-script-failures-4l90","comments_count":1,"public_reactions_count":0,"page_views_count":82,"published_timestamp":"2026-10-13T01:01:26Z","body_markdown":"A script that runs fine on your laptop and dies on the server is almost never a logic bug. It is an encoding bug that was always there, dormant, waiting for a machine with a different codepage. Windows makes this especially easy to trip over because the default encoding is not a constant — it is a property of the OS locale, and it leaks into every `open()` call, every `print()`, and every subprocess you spawn.\n\nHere is the failure mode, the mechanism, and the fixes that actually hold up in production.\n\n## The Two Machines\n\nThe scenario is common enough to be a template:\n\n- **Dev box:** Windows 11, \"Beta: Use Unicode UTF-8 for worldwide language support\" enabled. `sys.stdout.encoding` reports `utf-8`, `locale.getpreferredencoding()` reports `utf-8`.\n- **Production:** Chinese Windows Server 2016, default locale. Codepage 936 (GBK). `sys.stdout.encoding` reports `cp936`, `locale.getpreferredencoding()` reports `cp936`.\n\nSame Python version, same code. The dev box is in a UTF-8 bubble; the server is not. Everything that depends on the ambient encoding flips behavior between them.\n\n### What `open()` actually does\n\nWhen you call `open(path, \"w\")` without an `encoding` argument, Python uses `locale.getpreferredencoding(False)`. On the dev box that is `utf-8`. On the server that is `cp936`. This single default is the source of most cross-machine breakage.\n\n```python\n# This is not portable. The encoding is decided by the machine.\nwith open(\"report.txt\", \"w\") as f:\n    f.write(\"季度汇总\\n\")\n```\n\nOn the dev box, `report.txt` is UTF-8. On the server, it is GBK. If a downstream tool on either side assumes the other encoding, you get mojibake or a decode error. The file is byte-for-byte different depending on where it ran, and nothing in the code says so.\n\n### What `print()` actually does\n\n`print()` writes to `sys.stdout`, which is a `TextIOWrapper` whose encoding is derived from the console codepage. On the server, that is `cp936`. If you print a character that GBK cannot represent, you get:\n\n```plaintext\nUnicodeEncodeError: 'gbk' codec can't encode character '\\u2713' in position 0: illegal multibyte sequence\n```\n\nA checkmark, an emoji, a CJK extension character, a box-drawing glyph — any of these will crash a `print()` on the server and do nothing on the dev box. This is the failure that surfaces first because it is loud.\n\n## The Failure Cases\n\n### Case 1: `print()` crashes on the server\n\n```python\nprint(\"Done ✓\")  # fine on UTF-8 console, UnicodeEncodeError on cp936\n```\n\nThe traceback points at `print`, which makes it look like a stdout problem. It is. The console cannot represent `✓` in GBK, and Python raises rather than substituting.\n\n### Case 2: File paths get mangled\n\nThis one is quieter and worse. Windows paths are UTF-16 internally, but when they cross a codepage boundary — through a `.bat` file, an environment variable, or a subprocess argument — they get encoded and decoded using the active codepage. A path containing a character outside GBK, or a path that was written as UTF-8 in a bat file and read as GBK, comes back as garbage:\n\n```python\n# Path read from a config file that was saved as UTF-8\n# but the interpreter decoded it as cp936\npath = \"C:\\\\数据\\\\报表.csv\"   # becomes \"C:\\\\鏁版嵁\\\\鎶ヨ〃.csv\"\n```\n\nThe file exists. The path string does not match it. `FileNotFoundError` on a file you can see in Explorer.\n\n### Case 3: Bat files corrupt non-ASCII\n\nA `.bat` file with Chinese characters in it is interpreted using the console codepage at the time `cmd.exe` reads it. If the file is saved as UTF-8 (with or without BOM) and the console is cp936, `cmd.exe` misreads every multibyte character. `echo 开始` prints `寮€濮�`, and worse, any command built from those characters operates on the wrong path or argument.\n\n### Case 4: Subprocess output decoded with the wrong codec\n\n```python\nimport subprocess\nout = subprocess.run([\"some_tool.exe\"], capture_output=True).stdout\ntext = out.decode()  # uses locale.getpreferredencoding() -\u003e cp936 on server\n```\n\nIf `some_tool.exe` emits UTF-8, this decode produces mojibake or raises. If it emits GBK and you decode as UTF-8, same result in reverse. The subprocess inherits no encoding contract, so you have to impose one.\n\n## The Survival Rules\n\nThese five rules eliminate the entire class of bug. They are not stylistic preferences; each one closes a specific hole.\n\n### Rule 1: Always pass `encoding=` to `open()`\n\nNever rely on the locale default. Be explicit at every call site.\n\n```python\nimport io\n\n# Reading\nwith open(\"input.csv\", \"r\", encoding=\"utf-8\", newline=\"\") as f:\n    rows = f.read().splitlines()\n\n# Writing\nwith open(\"output.csv\", \"w\", encoding=\"utf-8\", newline=\"\") as f:\n    f.write(\"name,value\\n\")\n    f.write(\"季度,42\\n\")\n```\n\nFor CSV specifically, pass the encoding through to the `csv` module's file handle, not to the reader/writer:\n\n```python\nimport csv\n\nwith open(\"data.csv\", \"w\", encoding=\"utf-8\", newline=\"\") as f:\n    writer = csv.writer(f)\n    writer.writerow([\"symbol\", \"note\"])\n    writer.writerow([\"AAA\", \"正常\"])\n```\n\n`newline=\"\"` matters on Windows: without it, the `csv` module writes `\\r\\r\\n` because the text layer translates `\\n` and the writer already emits `\\r\\n`.\n\nIf you need to interoperate with Excel on a Chinese Windows box, `utf-8-sig` (UTF-8 with BOM) is the pragmatic choice for CSV — Excel will open it correctly without an import wizard:\n\n```python\nwith open(\"for_excel.csv\", \"w\", encoding=\"utf-8-sig\", newline=\"\") as f:\n    ...\n```\n\nBe aware this is a CSV/Excel concession. For logs and internal data, plain UTF-8 is correct.\n\n### Rule 2: Set `PYTHONIOENCODING=utf-8` in launchers\n\nSetting this environment variable forces `sys.stdin`, `sys.stdout`, and `sys.stderr` to UTF-8 regardless of the console codepage. In a bat launcher:\n\n```bat\n@echo off\nset PYTHONIOENCODING=utf-8\nset PYTHONUTF8=1\npython main.py\n```\n\n`PYTHONUTF8=1` enables UTF-8 mode, which additionally forces `locale.getpreferredencoding()` to return `utf-8`, making even the un-annotated `open()` calls behave consistently. Combining both is belt and suspenders, and it is cheap.\n\nThere is a catch: when stdout is redirected to a file or a pipe, the console codepage no longer matters and the encoding is whatever `PYTHONIOENCODING` says. When stdout is an interactive console, the console still has to *render* the characters. Setting `PYTHONIOENCODING=utf-8` stops the `UnicodeEncodeError`, but the console may show boxes for characters it cannot display. That is a cosmetic limitation of the console, not a data corruption — the bytes written to a redirected file are correct UTF-8.\n\nYou can verify what the interpreter sees:\n\n```python\nimport sys, locale\nprint(\"stdout:\", sys.stdout.encoding)\nprint(\"preferred:\", locale.getpreferredencoding(False))\nprint(\"utf8 mode:\", sys.flags.utf8_mode)\n```\n\nRun this on both machines before you debug anything else. It tells you immediately which regime you are in.\n\n### Rule 3: Wrap console prints in a GBK-safe fallback\n\nEven with `PYTHONIOENCODING` set, you cannot always control how your script is launched. A scheduled task, a service wrapper, or a colleague's `cmd` window may not set the variable. A defensive print wrapper costs little and never crashes:\n\n```python\nimport sys\n\ndef safe_print(*args, **kwargs):\n    \"\"\"Print without ever raising UnicodeEncodeError.\n\n    Falls back to a lossy ASCII representation when the target\n    stream cannot encode the text.\n    \"\"\"\n    stream = kwargs.get(\"file\", sys.stdout)\n    encoding = getattr(stream, \"encoding\", None) or \"ascii\"\n    text = \" \".join(str(a) for a in args)\n    try:\n        text.encode(encoding)\n    except (UnicodeEncodeError, LookupError):\n        text = text.encode(encoding, errors=\"replace\").decode(encoding)\n    print(text, **kwargs)\n```\n\n`safe_print(\"Done ✓\")` on a cp936 console prints `Done ?` instead of raising. The information about the checkmark is lost, but the script continues. For status output that is the right trade. For data you intend to parse later, do not print it — log it (Rule 4).\n\nIf you want to preserve more information, you can map known glyphs to ASCII equivalents before falling back:\n\n```python\n_FALLBACKS = {\"✓\": \"[OK]\", \"✗\": \"[FAIL]\", \"→\": \"-\u003e\", \"—\": \"-\"}\n\ndef safe_print(*args, **kwargs):\n    stream = kwargs.get(\"file\", sys.stdout)\n    encoding = getattr(stream, \"encoding\", None) or \"ascii\"\n    text = \" \".join(str(a) for a in args)\n    for ch, repl in _FALLBACKS.items():\n        if ch in text:\n            try:\n                text.encode(encoding)\n            except UnicodeEncodeError:\n                text = text.replace(ch, repl)\n    try:\n        text.encode(encoding)\n    except (UnicodeEncodeError, LookupError):\n        text = text.encode(encoding, errors=\"backslashreplace\").decode(encoding)\n    print(text, **kwargs)\n```\n\n### Rule 4: Write log files as UTF-8 regardless of console codepage\n\nLogs are data. They should not depend on the console. Configure logging with an explicit UTF-8 file handler and, if you also log to the console, let the console handler degrade gracefully.\n\n```python\nimport logging\nimport sys\n\nlogger = logging.getLogger(\"app\")\nlogger.setLevel(logging.INFO)\n\n# File handler: always UTF-8, independent of the console.\nfh = logging.FileHandler(\"app.log\", encoding=\"utf-8\")\nfh.setFormatter(logging.Formatter(\"%(asctime)s %(levelname)s %(message)s\"))\nlogger.addHandler(fh)\n\n# Console handler: keep it, but tolerate an unencodable console.\nch = logging.StreamHandler(sys.stdout)\nch.setFormatter(logging.Formatter(\"%(levelname)s %(message)s\"))\nlogger.addHandler(ch)\n\nlogger.info(\"处理完成,共 %d 条记录\", 128)\n```\n\nThe file always gets clean UTF-8 bytes. The console handler may still raise on a cp936 console if the message contains an unencodable character — which is why Rule 2 (set `PYTHONIOENCODING`) and Rule 3 (safe wrapper) matter. If you want the logging module itself to never raise, wrap the stream:\n\n```python\nimport io\n\nclass SafeStream(io.TextIOBase):\n    def __init__(self, raw):\n        self._raw = raw\n        self.encoding = getattr(raw, \"encoding\", None) or \"ascii\"\n\n    def write(self, s):\n        try:\n            s.encode(self.encoding)\n        except (UnicodeEncodeError, LookupError):\n            s = s.encode(self.encoding, errors=\"replace\").decode(self.encoding)\n        return self._raw.write(s)\n\n    def flush(self):\n        self._raw.flush()\n\nch = logging.StreamHandler(SafeStream(sys.stdout))\n```\n\nThis is the same idea as `safe_print`, applied at the stream level so every handler benefits.\n\n### Rule 5: Keep bat files ASCII-only\n\nThis is the rule people resist, and it is the one that saves the most pain. A `.bat` file is read by `cmd.exe` under the active console codepage. There is no reliable way to make a bat file containing non-ASCII text behave identically across codepages — the `chcp` command changes the console codepage but does not retroactively fix how `cmd.exe` parsed the bytes it already read from the file.\n\nKeep every `.bat` file pure ASCII. Move any non-ASCII string into a Python script or a UTF-8 data file that Python reads with an explicit encoding.\n\n```bat\n@echo off\nREM ASCII only. No Chinese characters in this file.\nset PYTHONIOENCODING=utf-8\nset PYTHONUTF8=1\npython \"%~dp0\\run_task.py\" %*\n```\n\nIf you must reference a path with non-ASCII characters, do it from Python, not from the bat file:\n\n```python\n# run_task.py\nfrom pathlib import Path\n\nbase = Path(r\"C:\\data\")          # ASCII in the source, safe\ntarget = base / \"季度报表.csv\"    # non-ASCII lives in Python source,\n                                  # which is UTF-8 by default (PEP 3120)\nwith open(target, \"r\", encoding=\"utf-8\", newline=\"\") as f:\n    ...\n```\n\nPython source files are UTF-8 by default since Python 3, so non-ASCII literals in `.py` files are safe. The bat file is the only place where the bytes are re-interpreted by a non-Python parser.\n\n## A Checklist\n\nBefore shipping a script that will run on a machine you do not control:\n\n1. Every `open()` has an explicit `encoding=`.\n2. Every subprocess capture is decoded with an explicit encoding, or you use `text=True, encoding=\"utf-8\"`.\n3. The launcher sets `PYTHONIOENCODING=utf-8` and `PYTHONUTF8=1`.\n4. Console output goes through a fallback that cannot raise.\n5. Log files are UTF-8, always.\n6. Bat files contain only ASCII.\n7. You ran the `sys.stdout.encoding` / `locale.getpreferredencoding` probe on both machines.\n\nThe pattern behind all of these is the same: stop letting the ambient locale decide anything. Encoding is a property of your program's data contract, not of the machine it happens to run on. Once you make it explicit, the dev box and the server stop disagreeing.\n\nMore notes like this ship every week on this site.","positive_reactions_count":0,"cover_image":null,"tag_list":["utf8","gbk","codepage","pythonioencoding"],"canonical_url":"https://www.hkcode.dpdns.org/en/posts/16.html","reading_time_minutes":8,"user":{"name":"mattleeee","username":"mattleeee","twitter_username":null,"github_username":"mattleeee","user_id":4160029,"website_url":null,"profile_image":"https://media2.dev.to/dynamic/image/width=640,height=640,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png","profile_image_90":"https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png"}},{"type_of":"article","id":4835153,"title":"The Silent Alert Bug: When Your Instant Push Exists but Never Fires","description":"Your alert formatter is probably fine. The problem is that nobody calls it.           The Symptom   A...","published":true,"published_at":"2026-10-12T01:01:35.084Z","slug":"the-silent-alert-bug-when-your-instant-push-exists-but-never-fires-3imc","path":"/mattleeee/the-silent-alert-bug-when-your-instant-push-exists-but-never-fires-3imc","url":"https://dev.to/mattleeee/the-silent-alert-bug-when-your-instant-push-exists-but-never-fires-3imc","comments_count":0,"public_reactions_count":0,"page_views_count":10,"published_timestamp":"2026-10-12T01:01:35Z","body_markdown":"Your alert formatter is probably fine. The problem is that nobody calls it.\n\n## The Symptom\n\nA user reports that instant push notifications stopped working. The daily report arrives on schedule. The formatting looks correct in the report. But the moment a position opens, nothing hits their phone.\n\nThe first instinct is to blame the transport layer: the push service, the API key, the network. You check credentials. You send a test push manually. It lands. So the pipeline is alive — which means the problem is upstream, in the code that's supposed to *invoke* the push.\n\nYou open the file. There's a function:\n\n```python\ndef format_instant_alert(position: dict) -\u003e str:\n    \"\"\"Build a single-line push message for a newly opened position.\"\"\"\n    side = position[\"side\"].upper()\n    symbol = position[\"symbol\"]\n    entry = position[\"entry_price\"]\n    qty = position[\"qty\"]\n    return f\"[{side}] {symbol} @ {entry} x{qty}\"\n```\n\nIt's clean, tested, and correct. It has been correct for months. Now grep for its call sites:\n\n```bash\ngrep -rn \"format_instant_alert\" --include=\"*.py\" .\n```\n\nOne hit: the definition itself. Zero call sites.\n\nThe function is dead code. The push never fires because nothing ever asks it to. The user's mental model — \"pushes are broken\" — is wrong. The pushes were never wired in. They exist as a well-formed corpse.\n\n## Four Paths, Four Comments, Zero Calls\n\nThe real scenario was worse than a single missing call. The system had four separate code paths for opening a position:\n\n1. Manual entry from the CLI.\n2. Scheduled strategy execution.\n3. Signal-driven entry from a webhook.\n4. Recovery/reconciliation after a restart.\n\nEach path had its own block of open-position logic — copy-pasted, then diverged. And each block carried a comment like this:\n\n```python\n# open the position, update state, push is integrated into the daily report\n```\n\nFour copies of the same sentence. Four copies of the same false assumption. The author of the original block believed the push was handled during daily report generation, so they didn't call the alert formatter. Every subsequent copy inherited the comment and the omission.\n\nThis is how \"handled elsewhere\" comments rot. The comment is a claim about code that lives somewhere else. It has no enforcement mechanism. When the \"elsewhere\" changes — or when it never actually handled the case — the comment stays, confidently lying. And because it's copy-pasted into four places, the lie propagates.\n\nThe daily report did not include instant alerts. It never had. The comment was aspirational, or it referred to a design that was abandoned. Nobody re-checked it, because comments are cheap and reading them feels like understanding.\n\n## Audit for Call Sites, Not Definitions\n\nThe core discipline is simple: **to find dead code, search for how a function is used, not where it's defined.**\n\nA definition search tells you the function exists. A call-site search tells you whether it matters. These are different questions, and conflating them is why dead code survives audits.\n\n### Step 1: Enumerate the suspects\n\nStart with the functions that *should* be firing but aren't. In an alerting system, that's typically the formatters and senders:\n\n```python\n# alerting.py\ndef format_instant_alert(position: dict) -\u003e str: ...\ndef format_daily_report(positions: list) -\u003e str: ...\ndef send_push(message: str) -\u003e bool: ...\ndef send_email(subject: str, body: str) -\u003e bool: ...\n```\n\n### Step 2: Count call sites with a script\n\nManual grep works for one function. For a whole module, automate it. This script counts references to each public function and flags the ones that only appear at their definition:\n\n```python\nimport ast\nimport subprocess\nfrom pathlib import Path\n\ndef defined_functions(path: str) -\u003e list[str]:\n    tree = ast.parse(Path(path).read_text(encoding=\"utf-8\"))\n    return [\n        node.name\n        for node in ast.walk(tree)\n        if isinstance(node, ast.FunctionDef) and not node.name.startswith(\"_\")\n    ]\n\ndef count_call_sites(name: str, root: str = \".\") -\u003e int:\n    \"\"\"Count lines mentioning `name` across the repo, excluding the definition.\"\"\"\n    result = subprocess.run(\n        [\"grep\", \"-rn\", \"--include=*.py\", name, root],\n        capture_output=True,\n        text=True,\n    )\n    hits = [line for line in result.stdout.splitlines() if line.strip()]\n    # A call site is any hit that is not a `def name(` line.\n    calls = [h for h in hits if f\"def {name}(\" not in h]\n    return len(calls)\n\nfor fn in defined_functions(\"alerting.py\"):\n    calls = count_call_sites(fn)\n    status = \"DEAD\" if calls == 0 else \"ok\"\n    print(f\"{status:5} {fn:30} call_sites={calls}\")\n```\n\nRun it and you get something like:\n\n```plaintext\nDEAD  format_instant_alert           call_sites=0\nok    format_daily_report            call_sites=2\nok    send_push                      call_sites=3\nok    send_email                     call_sites=1\n```\n\n`format_instant_alert` is the smoking gun. It's defined, it's importable, and nothing reaches it.\n\nThe grep-based approach is deliberately crude — it will count mentions in comments and strings, and it can't distinguish a real call from a reference. That's fine for a first pass. The goal is triage: find the functions with *zero* non-definition mentions. Those are unambiguously dead. For anything with a nonzero count, you verify by hand.\n\n### Step 3: Verify the call actually executes\n\nZero call sites is a clean verdict. A nonzero count is a lead, not a conclusion. A function can be \"called\" in a branch that never runs:\n\n```python\ndef open_position(order):\n    # ... state updates ...\n    if order.source == \"manual\":\n        send_push(format_instant_alert(order))  # only fires for manual entries\n```\n\nHere `format_instant_alert` has a call site, so the script marks it `ok`. But scheduled and webhook entries — the majority of real activity — never hit that branch. The function is reachable in theory and dead in practice.\n\nTo catch this, you need to know which *paths* call it, not just that a call exists. Trace each of the four open-position paths and confirm the push call is inside the code that actually runs for that path. If the call is guarded by a condition that your production traffic never satisfies, treat it as dead.\n\n## The Fix Discipline\n\nThe tempting fix is to add the missing call to the one path the user complained about. Don't. That leaves three paths still silent, and you'll be back here in a week when a webhook entry fails to notify.\n\n### Patch every occurrence in one pass\n\nFind every block that contains the same logic and fix all of them together. In this codebase, that meant four separate open-position blocks. The comment was the tell:\n\n```bash\ngrep -rn \"push is integrated into the daily report\" --include=\"*.py\" .\n```\n\nFour hits. Four blocks to patch. Each gets the same real call:\n\n```python\ndef open_position(order, state):\n    # ... state updates, persistence ...\n    state.add_position(order)\n\n    # Instant alert fires here, on every path, right after state is committed.\n    message = format_instant_alert(order)\n    ok = send_push(message)\n    if not ok:\n        log.warning(\"instant push failed for %s\", order[\"symbol\"])\n```\n\nRemove the stale comment while you're there. A comment that claims work happens elsewhere is worse than no comment — it actively suppresses the reader's suspicion.\n\n### Refactor to one path if you can\n\nFour copies of open-position logic is the root cause. The comment rot was just a symptom. If the paths can share a single function, do it:\n\n```python\ndef open_position(order, state, *, notify: bool = True):\n    validate(order)\n    state.add_position(order)\n    persist(state)\n    if notify:\n        send_push(format_instant_alert(order))\n```\n\nNow there is exactly one place where the push fires, and one place to audit. The next person who adds a fifth entry path calls this function and gets the alert for free.\n\n### Verify each call site executes\n\nAfter patching, re-run the audit script. `format_instant_alert` should show four call sites (or one, if you refactored). Then go further: prove each path actually reaches the call at runtime.\n\nThe cheapest proof is a log line at the call site and a test per path:\n\n```python\nimport logging\n\nlog = logging.getLogger(__name__)\n\ndef open_position(order, state, *, notify: bool = True):\n    validate(order)\n    state.add_position(order)\n    persist(state)\n    if notify:\n        log.info(\"dispatching instant alert for %s\", order[\"symbol\"])\n        send_push(format_instant_alert(order))\n```\n\nThen exercise all four paths in tests and assert the push was dispatched:\n\n```python\nfrom unittest.mock import patch\n\ndef test_all_entry_paths_dispatch_alert():\n    paths = [\n        {\"source\": \"manual\",   \"symbol\": \"AAA\", \"side\": \"long\", \"entry_price\": 10, \"qty\": 1},\n        {\"source\": \"schedule\", \"symbol\": \"BBB\", \"side\": \"long\", \"entry_price\": 20, \"qty\": 1},\n        {\"source\": \"webhook\",  \"symbol\": \"CCC\", \"side\": \"short\", \"entry_price\": 30, \"qty\": 1},\n        {\"source\": \"recovery\", \"symbol\": \"DDD\", \"side\": \"long\", \"entry_price\": 40, \"qty\": 1},\n    ]\n    for order in paths:\n        with patch(\"alerting.send_push\") as mock_push:\n            open_position(order, state=StubState())\n            assert mock_push.called, f\"no push for {order['source']}\"\n```\n\nThis test would have failed on all four paths before the fix. It fails loudly if someone later reintroduces a branch that skips the alert.\n\n## The General Pattern\n\nDead alert code is not a push problem. It's a code-path audit problem, and it shows up anywhere a side effect is assumed to be handled by someone else:\n\n- Alert formatters that are never invoked.\n- Cleanup routines that only run on one error branch.\n- Metrics that are defined but never incremented.\n- Retry handlers attached to a client that isn't the one in use.\n\nThe audit recipe is the same every time:\n\n1. **List the functions that should fire.** Search for call sites, not definitions.\n2. **Treat zero call sites as dead.** No exceptions, no \"it's probably called dynamically.\"\n3. **Treat guarded calls as suspicious.** Confirm the guard is satisfiable by real traffic.\n4. **Fix every copy in one pass.** If the same logic is duplicated, the bug is duplicated.\n5. **Prove execution.** A log line plus a per-path test beats reading the code and nodding.\n\nThe expensive part was never writing the formatter. It was the four comments that said the work was already done. Comments don't run. Call sites do.\n\nMore notes like this ship every week on this site.","positive_reactions_count":0,"cover_image":null,"tag_list":["deadcode","codepathaudit","alerts","grep"],"canonical_url":"https://www.hkcode.dpdns.org/en/posts/15.html","reading_time_minutes":6,"user":{"name":"mattleeee","username":"mattleeee","twitter_username":null,"github_username":"mattleeee","user_id":4160029,"website_url":null,"profile_image":"https://media2.dev.to/dynamic/image/width=640,height=640,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png","profile_image_90":"https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png"}},{"type_of":"article","id":4830785,"title":"Notification Deduplication: A Five-Minute Title Window","description":"Alerts that fire twice for the same event are worse than alerts that never fire at all. The first...","published":true,"published_at":"2026-10-11T01:01:15.146Z","slug":"notification-deduplication-a-five-minute-title-window-206f","path":"/mattleeee/notification-deduplication-a-five-minute-title-window-206f","url":"https://dev.to/mattleeee/notification-deduplication-a-five-minute-title-window-206f","comments_count":0,"public_reactions_count":0,"page_views_count":60,"published_timestamp":"2026-10-11T01:01:15Z","body_markdown":"Alerts that fire twice for the same event are worse than alerts that never fire at all. The first duplicate teaches you the system is noisy; the tenth teaches you to swipe notifications away without reading them. Once that habit forms, a real alert gets the same treatment.\n\nThe usual cause is not a broken alert rule. It is retries and overlapping schedulers. A cron job restarts mid-run, the previous process is still alive, both evaluate the same condition, both push. Or a task queue retries after a timeout while the original request actually succeeded. Or you run the same check from two machines during a deploy. Each path is individually correct, and together they produce two notifications for one event.\n\nDeduplication belongs at the push boundary, not in the detection logic. Detection rules are where you want redundancy; the notification layer is where you want exactly-once behavior. This post covers a small, dependency-free dedup cache that enforces a five-minute window keyed on the notification title, with an explicit bypass for signals that must never be suppressed.\n\n## The dedup contract\n\nKeep the rule short enough to state in one sentence:\n\n\u003e Before pushing a notification, look up its normalized title in a JSON cache. If the same title was pushed within the last five minutes, skip the push and log the suppression. Otherwise, push and record the timestamp.\n\nTwo properties matter:\n\n- **Idempotency by title.** Two calls with the same title inside the window produce one push.\n- **Fail-open.** If the cache is unreadable or corrupt, push anyway. A duplicate is annoying; a swallowed alert is a bug you find out about too late.\n\nThe cache is deliberately tiny — one file, one dict, no database. This is not an event log. It is a short-lived suppression table, and anything older than the window is garbage.\n\n## Cache file layout\n\nThe file is a flat JSON object mapping normalized titles to Unix timestamps:\n\n```json\n{\n  \"disk_usage_high:/var\": 1730541123.44,\n  \"feed_stale:BTC-USD\": 1730541401.02,\n  \"worker_queue_backlog\": 1730539807.91\n}\n```\n\nA few decisions worth being explicit about:\n\n- **Title as key, timestamp as value.** No arrays, no per-key history. You only ever need the most recent push time to answer \"was this within the window?\"\n- **Unix epoch floats.** Timezone-free, trivially comparable, and easy to prune. If you store ISO strings you have to parse them back, and parsing is where bugs live.\n- **One file, not one file per title.** A per-title layout turns a read into a directory scan and makes atomic updates harder. With a few hundred keys, a single file is microseconds.\n- **Prune on write.** Entries older than the window are dropped when the file is rewritten. The file stays bounded by the number of distinct titles active in the last five minutes, which is small.\n\nIf you want a durable audit trail of what was suppressed, write that to your normal log sink. Do not grow the cache into a database — the moment it becomes stateful infrastructure, it becomes a thing that can fail and block pushes.\n\n## Atomic writes with tempfile and os.replace\n\nThe cache is read and written by processes that may run concurrently. A naive `open(path, \"w\")` truncates the file before writing, so a reader that lands in that gap sees an empty file. Worse, a crash mid-write leaves a half-written JSON document that fails to parse on the next run.\n\nThe fix is the standard write-to-temp-then-rename pattern:\n\n```python\nimport json\nimport os\nimport tempfile\n\ndef _write_cache(path: str, data: dict) -\u003e None:\n    directory = os.path.dirname(path) or \".\"\n    fd, tmp_path = tempfile.mkstemp(dir=directory, prefix=\".dedup-\", suffix=\".tmp\")\n    try:\n        with os.fdopen(fd, \"w\", encoding=\"utf-8\") as fh:\n            json.dump(data, fh)\n            fh.flush()\n            os.fsync(fh.fileno())\n        os.replace(tmp_path, path)\n    except BaseException:\n        # Clean up the temp file if anything went wrong before the rename.\n        try:\n            os.unlink(tmp_path)\n        except FileNotFoundError:\n            pass\n        raise\n```\n\nWhy this works:\n\n- `tempfile.mkstemp` creates the temp file in the **same directory** as the target. `os.replace` is atomic only within a filesystem, so a temp file in `/tmp` renamed onto a path in `/var/lib` can degrade to a copy — or fail outright with `EXDEV`.\n- `flush` plus `os.fsync` pushes bytes to disk before the rename. Without the fsync, a crash can leave you with a renamed file whose contents were never durable.\n- `os.replace` is atomic on POSIX and on Windows. Readers see either the old file or the new one, never a truncated intermediate.\n- On any exception before the rename, the temp file is removed so you do not accumulate `.dedup-*.tmp` litter.\n\nThis pattern costs one extra file creation per write. For a cache that is written at most a few times per minute, that is irrelevant.\n\n## The dedup check\n\nPutting it together: read, decide, push, record. The read path must tolerate a missing or corrupt file by returning an empty dict.\n\n```python\nimport json\nimport logging\nimport os\nimport time\n\nlog = logging.getLogger(__name__)\n\nWINDOW_SECONDS = 300.0\nCACHE_PATH = \"/var/lib/alerts/dedup.json\"\n\ndef _load_cache(path: str) -\u003e dict:\n    try:\n        with open(path, \"r\", encoding=\"utf-8\") as fh:\n            data = json.load(fh)\n    except FileNotFoundError:\n        return {}\n    except (json.JSONDecodeError, OSError) as exc:\n        # Fail open: a broken cache must not block notifications.\n        log.warning(\"dedup cache unreadable (%s); treating as empty\", exc)\n        return {}\n    if not isinstance(data, dict):\n        log.warning(\"dedup cache has unexpected shape; treating as empty\")\n        return {}\n    return data\n\ndef _prune(data: dict, now: float) -\u003e dict:\n    cutoff = now - WINDOW_SECONDS\n    return {k: v for k, v in data.items()\n            if isinstance(v, (int, float)) and v \u003e= cutoff}\n\ndef should_push(title: str, *, urgent: bool = False,\n                cache_path: str = CACHE_PATH,\n                window: float = WINDOW_SECONDS) -\u003e bool:\n    \"\"\"Return True if the notification should be pushed.\"\"\"\n    if urgent:\n        log.info(\"dedup bypass (urgent): %s\", title)\n        return True\n\n    now = time.time()\n    data = _load_cache(cache_path)\n    last = data.get(title)\n\n    if isinstance(last, (int, float)) and (now - last) \u003c window:\n        log.info(\"dedup suppressed: %s (last push %.1fs ago)\", title, now - last)\n        return False\n\n    data = _prune(data, now)\n    data[title] = now\n    _write_cache(cache_path, data)\n    return True\n```\n\nCall it right before the transport:\n\n```python\ndef push(title: str, body: str, *, urgent: bool = False) -\u003e None:\n    if not should_push(title, urgent=urgent):\n        return\n    send_to_device(title, body)\n```\n\nThe suppression is logged with the age of the last push, which is what you want when someone asks why they did not get an alert. \"Suppressed 42 seconds after the previous push of the same title\" is a complete answer.\n\n### A note on the check-then-write race\n\nTwo processes can both read the cache, both see no recent entry, and both push. The window narrows the race but does not eliminate it. If you need strict exactly-once across processes, put an advisory lock around the read-modify-write:\n\n```python\nimport fcntl\n\ndef with_lock(path: str):\n    lock_path = path + \".lock\"\n    fh = open(lock_path, \"w\")\n    fcntl.flock(fh, fcntl.LOCK_EX)\n    return fh  # caller closes to release\n```\n\nOn Windows, use `msvcrt.locking` or a named mutex. In practice, most alerting pipelines are single-process or serialize pushes through a queue, and the window alone is enough. Add the lock only when you have measured the duplicate rate.\n\n## Why the window keys on the normalized title\n\nThe single most important design choice is what you hash on. Keying on the full notification body is tempting — it is the thing you actually send — but it breaks dedup in exactly the cases you care about.\n\nConsider a price or metric alert. The body almost always contains the current value:\n\n```plaintext\nTitle: Disk usage high: /var\nBody:  /var is at 91.4% (threshold 90%)\n```\n\nThe next check two minutes later produces `91.7%`. Different body, same condition, second notification. The user sees two alerts for one ongoing problem. Keying on the body means dedup only catches byte-identical retries, which are the easy case you could handle with a request ID anyway.\n\nThe title, by contrast, is the *identity of the condition*. Normalize it and use that:\n\n```python\nimport re\n\ndef normalize_title(title: str) -\u003e str:\n    t = title.strip().lower()\n    t = re.sub(r\"\\s+\", \" \", t)\n    return t\n```\n\nNormalization matters because titles are often assembled from templates and may differ in case or whitespace between code paths. `\"Disk usage high: /var\"` and `\"disk usage high:  /var\"` are the same condition and must collapse to one key.\n\nThe rule that follows: **put the stable identity in the title and the volatile detail in the body.** If a title embeds a timestamp, a hostname that changes per replica, or a fluctuating value, dedup will not work — and the fix is to change the title, not to loosen the window.\n\nIf you genuinely need sub-condition granularity, make the condition part of the title deliberately: `disk_usage_high:/var` and `disk_usage_high:/home` are separate keys, which is correct. The key should be exactly as specific as the thing a human would consider \"the same alert.\"\n\n## Bypassing the window for urgent signals\n\nA five-minute window is right for most alerts and wrong for a few. If a process is crash-looping, or a safety condition trips, you do not want the second occurrence suppressed because the first happened ninety seconds ago.\n\nThe `urgent` flag above is the escape hatch. Two rules keep it from eroding the whole design:\n\n1. **Urgency is a property of the alert rule, not of the caller's mood.** Decide at definition time which titles are urgent, and keep that list short. If everything is urgent, nothing is deduplicated.\n2. **Urgent pushes still update the cache.** An urgent push writes its timestamp like any other, so a subsequent non-urgent push of the same title inside the window is still suppressed. You get the urgent signal immediately, and you do not get a redundant routine one a minute later.\n\nA common refinement is a *shorter* window for urgent titles rather than a full bypass — say thirty seconds instead of five minutes — so that a crash loop produces at most two notifications per minute instead of one per restart. The implementation is the same; you just look up the window per title:\n\n```python\nURGENT_WINDOW_SECONDS = 30.0\n\ndef window_for(title: str, urgent: bool) -\u003e float:\n    return URGENT_WINDOW_SECONDS if urgent else WINDOW_SECONDS\n```\n\nPass that value into `should_push` instead of the fixed `window` default.\n\n## Operating notes\n\n- **Where to put the file.** A local path with a stable directory, e.g. `/var/lib/alerts/dedup.json`. If you run multiple hosts that each push independently, each needs its own cache, or you need a shared store. A local file per host is usually correct: duplicate suppression across hosts is a different problem, solved by routing pushes through one service.\n- **Permissions.** The file contains alert titles, which may leak hostnames or symbols. Restrict it to the service user.\n- **Monitoring the cache.** Log the suppression count. A sudden jump in suppressions means either a retry storm or a window that is too wide. Both are worth knowing.\n- **Testing.** Inject `now` instead of calling `time.time()` inside the function so you can test window boundaries deterministically. The version above takes `time.time()` inline for brevity; in production, pass a clock.\n\nThe whole mechanism is under sixty lines, has no dependencies, and removes the most common source of alert fatigue. The window is a blunt instrument, but blunt and predictable beats clever and occasionally silent.\n\nMore notes like this ship every week on this site.","positive_reactions_count":0,"cover_image":null,"tag_list":["deduplication","idempotency","notifications","timewindow"],"canonical_url":"https://www.hkcode.dpdns.org/en/posts/14.html","reading_time_minutes":7,"user":{"name":"mattleeee","username":"mattleeee","twitter_username":null,"github_username":"mattleeee","user_id":4160029,"website_url":null,"profile_image":"https://media2.dev.to/dynamic/image/width=640,height=640,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png","profile_image_90":"https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png"}},{"type_of":"article","id":4826000,"title":"Firewall Forensics: PAT Sessions, NAT44 Hit Counters and the Missing UDP Rule","description":"A GB28181 camera was sending heartbeats. The SIP server saw them. The firewall session table...","published":true,"published_at":"2026-10-10T01:01:20.690Z","slug":"firewall-forensics-pat-sessions-nat44-hit-counters-and-the-missing-udp-rule-3kho","path":"/mattleeee/firewall-forensics-pat-sessions-nat44-hit-counters-and-the-missing-udp-rule-3kho","url":"https://dev.to/mattleeee/firewall-forensics-pat-sessions-nat44-hit-counters-and-the-missing-udp-rule-3kho","comments_count":0,"public_reactions_count":0,"page_views_count":30,"published_timestamp":"2026-10-10T01:01:20Z","body_markdown":"A GB28181 camera was sending heartbeats. The SIP server saw them. The firewall session table confirmed the outbound translation. And yet inbound INVITEs never arrived — the NAT44 hit counter for UDP 7100 sat at exactly 0 while TCP 7100 on the same port showed 13 hits. That asymmetry is the whole story, and it's the kind of clue that gets buried under \"the network is broken\" tickets.\n\nHere's how to read that evidence, why it matters, and how to turn a vague complaint into a reproducible pass/fail list.\n\n## The setup\n\nGB28181 is a Chinese video surveillance standard built on SIP. Devices register to a platform, then receive INVITEs to start streaming. The signaling runs over UDP 7100 in this deployment. A camera at `10.0.13.200` sits behind a firewall doing PAT (Port Address Translation, the overloaded form of NAT44). The platform sits somewhere upstream, reachable only through a public IP.\n\nThe complaint: \"the camera registers but we can't pull streams.\" Classic. Registration is outbound, streaming setup is inbound, and everything in between is where the bodies are buried.\n\n## Reading the session table\n\nThe first useful artifact is the firewall's session table. On most enterprise firewalls it looks something like this:\n\n```plaintext\nProtocol  Src              Sport  Dst              Dport  TransSrc         TransSport  State\nUDP       10.0.13.200      7100   203.0.113.40     7100   198.51.100.77    20250       ACTIVE\nTCP       10.0.13.200      7100   203.0.113.40     7100   198.51.100.77    20251       ESTABLISHED\n```\n\nTwo things to notice.\n\nFirst, the outbound heartbeat from `10.0.13.200:7100` was translated to `198.51.100.77:20250`. That's PAT doing its job. The source port changed from 7100 to 20250 — the public side sees a different port entirely. This is normal and expected; the camera has no idea its port was rewritten.\n\nSecond, the session is `ACTIVE`, which means the firewall has state for it. Heartbeats keep it alive. So the NAT rule matched, the translation happened, and the packet left the building.\n\nNow the counter that actually matters:\n\n```plaintext\nNAT44 policy hit counters:\n  UDP 7100: 0\n  TCP 7100: 13\n```\n\nZero. On UDP. Thirteen on TCP.\n\nThis is the moment where a lot of troubleshooting goes wrong. People see \"0 hits\" and assume the NAT rule is broken. It isn't. The rule matched — we just watched it translate a packet. The counter is telling us something narrower and more useful: **inbound** packets on UDP 7100 never reached the NAT policy at all.\n\n## NAT and security policy are different layers\n\nThis trips up experienced engineers, so it's worth being explicit.\n\nA firewall evaluates packets in order:\n\n1. **Ingress interface / zone assignment**\n2. **Security policy** (allow/deny between zones, possibly with application inspection)\n3. **NAT policy** (translation, including destination NAT for inbound)\n4. **Session creation and forwarding**\n\nInbound packets hit the security policy first, then NAT. If a packet is dropped by the security policy, or by an upstream filter that runs before the firewall even sees it, the NAT policy never increments its counter. The counter isn't a measure of \"did the rule work\" — it's a measure of \"did any packet reach this rule.\"\n\nSo `UDP 7100 hits = 0` means one of:\n\n- The packet never arrived at the firewall.\n- The packet arrived but was dropped by security policy, ASPF, or a DDoS/blacklist mechanism before NAT.\n- The packet arrived on a different interface or zone than expected.\n\nIn this case, the outbound session proves the NAT rule is correct and functional. The zero counter on inbound proves the problem is **upstream of the NAT decision**. The two most likely culprits:\n\n- **The ISP modem has no UDP mapping.** If the modem is doing its own NAT (double NAT), it needs a port-forward for UDP 7100 to reach the firewall. Many consumer and SMB modems handle TCP port forwards fine but silently drop UDP, or the forward was never created for UDP.\n- **ASPF or a DDoS blacklist is eating it.** Application-layer gateways that inspect SIP sometimes drop packets that don't fit expected patterns. DDoS protection lists can flag UDP 7100 as suspicious traffic, especially from an unrecognized source.\n\nBoth failure modes produce the same signature: the packet dies before NAT, so the counter stays at zero.\n\n## Why \"telnet works\" proves nothing\n\nEvery ticket has a version of this line: \"But I can telnet to port 7100 from outside, so the port is open.\"\n\nNo. Telnet is TCP. TCP 7100 has 13 hits — of course it works. The camera doesn't speak TCP on 7100; it speaks UDP. A TCP success tells you the ISP modem forwards TCP, the firewall allows TCP, and the NAT policy for TCP is intact. It tells you nothing about UDP.\n\nThis is the single most common false positive in network troubleshooting. TCP and UDP are separate protocols with separate connection tracking, separate NAT mappings, and separate policy rules. A port is not \"open\" or \"closed\" — a (protocol, port) pair is.\n\nThe fix is to stop testing with telnet and start testing with a probe that actually speaks the protocol in question.\n\n## A TCP + UDP port probe\n\nHere's a script that turns \"the port is broken\" into a pass/fail matrix. It runs from outside the firewall and tests both TCP and UDP on the same port, then reports which combinations succeed.\n\n```python\n#!/usr/bin/env python3\n\"\"\"Probe TCP and UDP reachability on a target port.\n\nReports a pass/fail matrix so a vague complaint becomes a\nreproducible result. Run from a host outside the firewall.\n\"\"\"\n\nimport socket\nimport argparse\nimport time\n\n\ndef probe_tcp(host: str, port: int, timeout: float = 3.0) -\u003e bool:\n    \"\"\"Return True if a TCP connection completes.\"\"\"\n    try:\n        with socket.create_connection((host, port), timeout=timeout):\n            return True\n    except (socket.timeout, ConnectionRefusedError, OSError):\n        return False\n\n\ndef probe_udp(host: str, port: int, timeout: float = 3.0,\n              payload: bytes = b\"\\r\\n\\r\\n\") -\u003e str:\n    \"\"\"Send a UDP datagram and classify the response.\n\n    Returns one of: 'reply', 'icmp-unreachable', 'timeout'.\n    A timeout is ambiguous: the packet may have been dropped, or\n    the service may simply not reply to this payload. An ICMP\n    port-unreachable means the packet reached a host and was\n    rejected at the transport layer.\n    \"\"\"\n    sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)\n    sock.settimeout(timeout)\n    try:\n        sock.sendto(payload, (host, port))\n        try:\n            data, _ = sock.recvfrom(4096)\n            return \"reply\" if data else \"reply\"\n        except socket.timeout:\n            return \"timeout\"\n        except ConnectionRefusedError:\n            return \"icmp-unreachable\"\n    finally:\n        sock.close()\n\n\ndef main() -\u003e None:\n    ap = argparse.ArgumentParser(description=\"TCP/UDP port probe\")\n    ap.add_argument(\"host\")\n    ap.add_argument(\"port\", type=int)\n    ap.add_argument(\"--timeout\", type=float, default=3.0)\n    args = ap.parse_args()\n\n    tcp_ok = probe_tcp(args.host, args.port, args.timeout)\n    udp_result = probe_udp(args.host, args.port, args.timeout)\n\n    print(f\"Target: {args.host}:{args.port}\")\n    print(f\"  TCP: {'PASS' if tcp_ok else 'FAIL'}\")\n    print(f\"  UDP: {udp_result.upper()}\")\n\n    if tcp_ok and udp_result == \"timeout\":\n        print(\"\\nDiagnosis: TCP reaches the target, UDP does not.\")\n        print(\"Check the ISP modem for a UDP port forward, and\")\n        print(\"inspect ASPF / DDoS policy on the firewall.\")\n\n\nif __name__ == \"__main__\":\n    main()\n```\n\nRun it against the public IP:\n\n```plaintext\n$ python3 probe.py 198.51.100.77 7100\nTarget: 198.51.100.77:7100\n  TCP: PASS\n  UDP: TIMEOUT\n\nDiagnosis: TCP reaches the target, UDP does not.\nCheck the ISP modem for a UDP port forward, and\ninspect ASPF / DDoS policy on the firewall.\n```\n\nThat output is a ticket-closer. It's specific, reproducible, and it points at the right layer.\n\nOne caveat on UDP probing: a timeout is ambiguous. The packet may have been dropped, or the service may simply not reply to an arbitrary payload. If you can, send a payload the service actually understands — a real SIP OPTIONS request for GB28181, for example — so a reply confirms reachability rather than just absence of an error.\n\n## A SIP-aware variant\n\nFor GB28181 specifically, a bare UDP probe may not get a reply even when the path is open. Send a real SIP OPTIONS and watch for a response:\n\n```python\nimport socket\n\ndef sip_options_probe(host: str, port: int, timeout: float = 5.0) -\u003e bool:\n    \"\"\"Send a SIP OPTIONS request and wait for any response.\"\"\"\n    msg = (\n        f\"OPTIONS sip:{host} SIP/2.0\\r\\n\"\n        f\"Via: SIP/2.0/UDP 0.0.0.0:{port};branch=z9hG4bK-probe\\r\\n\"\n        f\"From: \u003csip:probe@localhost\u003e;tag=probe\\r\\n\"\n        f\"To: \u003csip:{host}\u003e\\r\\n\"\n        f\"Call-ID: probe-{int(time.time())}\\r\\n\"\n        f\"CSeq: 1 OPTIONS\\r\\n\"\n        f\"Max-Forwards: 70\\r\\n\"\n        f\"Content-Length: 0\\r\\n\\r\\n\"\n    ).encode()\n\n    sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)\n    sock.settimeout(timeout)\n    try:\n        sock.sendto(msg, (host, port))\n        data, _ = sock.recvfrom(4096)\n        return b\"SIP/2.0\" in data\n    except socket.timeout:\n        return False\n    finally:\n        sock.close()\n```\n\nIf this returns `False` while TCP 7100 is open, you've confirmed the UDP path is broken before it reaches the firewall. If it returns `True`, the path is fine and the problem is elsewhere — the platform's INVITE routing, the camera's registration binding, or the firewall's SIP ALG mangling headers.\n\n## Reading the counters correctly\n\nThe general lesson: **a zero hit counter on a NAT rule doesn't mean the rule is wrong — it means no packet reached it.** Compare the counter against the session table. If sessions exist but the counter is zero, the traffic that created those sessions never traversed the rule in the direction you're checking.\n\nBuild a habit of asking three questions for any \"port is broken\" report:\n\n1. **What does the session table show?** Is there an active session? What's the translation?\n2. **What do the policy hit counters show, per protocol?** A TCP hit count next to a zero UDP hit count is a specific diagnosis, not a coincidence.\n3. **What does a protocol-aware probe show from outside?** Telnet is not a UDP test. Ever.\n\nThe GB28181 case resolved when the ISP modem's UDP 7100 forward was added. The firewall was never the problem — the NAT rule had been translating heartbeats correctly the whole time. The counter was telling the truth; it just took a while to read it.\n\nMore notes like this ship every week on this site.\n\n---\n\n## Daily Picks\n\n\u003e The following pairs are selected from the multi-timeframe trend scanner (Gate.io futures) and are for technical-analysis study only — not investment advice.\n\u003e Data updated: 2026-10-09 08:46:31\n\n### Short\n\n| Pair | Signal | Price | Take Profit | Stop Loss | R/R |\n|------|--------|-------|-------------|-----------|-----|\n| COOL | 短平快做空 | $0.0015 | $0.0014 | $0.0015 | 1:1.2 |\n\n_1 picks selected. Scanner runs every 15 minutes._","positive_reactions_count":0,"cover_image":null,"tag_list":["pat","nat44","sessiontable","udp7100"],"canonical_url":"https://www.hkcode.dpdns.org/en/posts/12.html","reading_time_minutes":7,"user":{"name":"mattleeee","username":"mattleeee","twitter_username":null,"github_username":"mattleeee","user_id":4160029,"website_url":null,"profile_image":"https://media2.dev.to/dynamic/image/width=640,height=640,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png","profile_image_90":"https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png"}},{"type_of":"article","id":4825999,"title":"Wiring WeChat Push into Python Automation with PushPlus","description":"PushPlus is the shortest path from a Python script to your phone. One HTTP POST with a token, a...","published":true,"published_at":"2026-10-10T01:01:17.338Z","slug":"wiring-wechat-push-into-python-automation-with-pushplus-fl","path":"/mattleeee/wiring-wechat-push-into-python-automation-with-pushplus-fl","url":"https://dev.to/mattleeee/wiring-wechat-push-into-python-automation-with-pushplus-fl","comments_count":1,"public_reactions_count":0,"page_views_count":64,"published_timestamp":"2026-10-10T01:01:17Z","body_markdown":"PushPlus is the shortest path from a Python script to your phone. One HTTP POST with a token, a title, and a markdown body, and the message lands in WeChat. No app to build, no bot framework, no SMTP relay that gets flagged as spam. For unattended jobs — data collection, model retraining, portfolio snapshots — that single channel is usually enough.\n\nThis post covers the setup end to end: getting the token, a send function that survives network hiccups, naming conventions that keep dozens of scripts distinguishable, and the templates I reuse for daily reports versus alerts.\n\n## Getting the token\n\nRegister at `pushplus.plus` (or the Chinese mirror, `pushplus.plus` routes there too) with WeChat scan login. The account is bound to your WeChat, which is what makes the push arrive as a service message.\n\nAfter login, open **Send Message** → **One-on-One Message**. The page shows a 32-character token. Treat it like a password: it authorizes anyone who holds it to send messages to your WeChat. Store it in an environment variable, not in source code.\n\n```bash\n# .env or your shell profile\nexport PUSHPLUS_TOKEN=\"a1b2c3d4e5f6...\"\n```\n\n```python\nimport os\n\nTOKEN = os.environ[\"PUSHPLUS_TOKEN\"]\n```\n\nA free account allows a limited number of pushes per day. For a handful of jobs that run a few times daily, that ceiling is never a problem. If you push per-row records or log every heartbeat, you will hit it — batch instead.\n\n## A send function that does not lie to you\n\nThe naive version is `requests.post(url, json=payload)` and done. That fails in three ways that matter for unattended jobs:\n\n1. No timeout. A hung connection blocks the job forever.\n2. No status check. PushPlus returns HTTP 200 even when the token is invalid; the error is in the JSON body.\n3. No retry. A transient DNS failure kills the notification for the one run that needed it.\n\nHere is the version I actually use:\n\n```python\nimport logging\nimport time\n\nimport requests\n\nPUSHPLUS_URL = \"https://www.pushplus.plus/send\"\nTOKEN = os.environ[\"PUSHPLUS_TOKEN\"]\n\nlogger = logging.getLogger(__name__)\n\n\ndef push(\n    title: str,\n    content: str,\n    *,\n    template: str = \"markdown\",\n    token: str = TOKEN,\n    timeout: float = 10.0,\n    retries: int = 2,\n) -\u003e bool:\n    \"\"\"Send a WeChat push via PushPlus. Returns True on confirmed delivery.\"\"\"\n    payload = {\n        \"token\": token,\n        \"title\": title,\n        \"content\": content,\n        \"template\": template,\n    }\n\n    for attempt in range(retries + 1):\n        try:\n            resp = requests.post(PUSHPLUS_URL, json=payload, timeout=timeout)\n            resp.raise_for_status()\n            body = resp.json()\n        except requests.RequestException as exc:\n            logger.warning(\"push attempt %d failed: %s\", attempt + 1, exc)\n            if attempt \u003c retries:\n                time.sleep(2 ** attempt)\n                continue\n            return False\n\n        if body.get(\"code\") == 200:\n            return True\n\n        # Business-level error: bad token, quota exceeded, etc.\n        logger.error(\"push rejected: %s\", body.get(\"msg\"))\n        return False\n\n    return False\n```\n\nThree details worth calling out.\n\n**Check `code`, not just the HTTP status.** PushPlus wraps everything in `{\"code\": 200, \"msg\": \"请求成功\", \"data\": \"...\"}`. An expired token returns HTTP 200 with `code: 401`. If you only check `raise_for_status()`, you will believe a dead token is working.\n\n**Exponential backoff on network errors only.** Retrying a business error (bad token, quota) just burns quota. Retrying a socket timeout is worth it. The code above separates the two paths.\n\n**Return a boolean.** Callers need to know whether the push succeeded. A job that cannot report its own failure should at least be able to log that the reporting failed.\n\n## Title conventions: prefix everything\n\nOnce you have five or six scripts pushing to the same WeChat thread, the notification list becomes a wall of text. The fix is a fixed prefix per system, so a glance at the lock screen tells you what fired and whether it matters.\n\n```plaintext\n[QUANT] Daily NAV — 2024-06-14\n[SCRAPE] Job failed — 3 retries exhausted\n[CLEAN] Weekly dedup done — 12,340 rows\n[MONITOR] Disk usage 91% on host-02\n```\n\nThe pattern is `[SYSTEM] Short description — key metric or timestamp`. The bracket tag is scannable; the em dash separates the event from its payload. Keep the tag under eight characters and uppercase it.\n\nI centralize this in a tiny wrapper so no script invents its own format:\n\n```python\ndef notify(system: str, subject: str, body: str) -\u003e bool:\n    title = f\"[{system.upper()}] {subject}\"\n    return push(title, body)\n```\n\nCall sites stay clean:\n\n```python\nnotify(\"quant\", \"Daily NAV — 2024-06-14\", report_md)\nnotify(\"scrape\", \"Job failed — 3 retries exhausted\", traceback_md)\n```\n\nTwo rules keep this useful. First, never put the variable part of the message in the prefix — the prefix is for filtering, the suffix is for reading. Second, keep the tag vocabulary small. If you have fifteen systems, group them into four or five tags (`QUANT`, `DATA`, `OPS`, `ALERT`) and put the specific name in the subject.\n\n## Markdown templates: reports versus alerts\n\nPushPlus renders `template: \"markdown\"` natively in WeChat. That means headers, bold, tables, and code blocks all render. Use it — a wall of plain text is harder to parse on a phone than a structured card.\n\nThe mistake is using one template for everything. A daily report and a failure alert have opposite reading patterns. The report is scanned for the numbers; the alert is scanned for the cause and the action.\n\n### Daily report template\n\nReports are read on a schedule, often half-awake. Front-load the numbers, push detail below.\n\n```python\ndef daily_report(date: str, metrics: dict, notes: str = \"\") -\u003e str:\n    rows = \"\\n\".join(\n        f\"| {k} | {v} |\" for k, v in metrics.items()\n    )\n    return f\"\"\"## Daily Report — {date}\n\n| Metric | Value |\n| --- | --- |\n{rows}\n\n**Notes**\n\n{notes or \"_none_\"}\n\"\"\"\n```\n\nThe `##` header renders as a section title in WeChat. The table aligns values. The notes section is where you put a one-liner about anything unusual — an empty notes field is a signal in itself.\n\n### Alert template\n\nAlerts are read under stress. The first line must answer \"what broke,\" the second \"why,\" the third \"what I should do.\" No preamble.\n\n```python\ndef alert_body(\n    what: str,\n    detail: str,\n    action: str,\n    traceback_text: str | None = None,\n) -\u003e str:\n    body = f\"\"\"**What:** {what}\n\n**Detail:** {detail}\n\n**Action:** {action}\n\"\"\"\n    if traceback_text:\n        body += f\"\\n```\\n{traceback_text[-1500:]}\\n```\\n\"\n    return body\n```\n\nThe traceback is truncated to the last 1500 characters. The tail is where the actual exception lives; the head is usually framework noise. Truncating keeps the push readable and avoids hitting message size limits.\n\nUse alerts sparingly. If every alert fires daily, none of them are alerts. Reserve the alert template for conditions that require a human decision — a failed job that will not self-heal, a threshold breach, a resource exhaustion.\n\n## Every unattended job ends with a push\n\nThis is the rule that matters most. **Silence is not success.**\n\nA cron job that runs, does nothing visible, and exits is indistinguishable from a cron job that never started, crashed before logging, or was silently disabled by the scheduler. The only way to know is to make the job announce itself — every run, both outcomes.\n\n```python\nimport sys\nimport traceback\n\n\ndef main() -\u003e None:\n    try:\n        result = run_pipeline()\n    except Exception:\n        notify(\n            \"quant\",\n            \"Pipeline failed\",\n            alert_body(\n                what=\"run_pipeline raised\",\n                detail=\"unhandled exception\",\n                action=\"check logs on host-02\",\n                traceback_text=traceback.format_exc(),\n            ),\n        )\n        sys.exit(1)\n    else:\n        notify(\n            \"quant\",\n            f\"Pipeline done — {result['rows']} rows\",\n            daily_report(result[\"date\"], result[\"metrics\"], result[\"notes\"]),\n        )\n\n\nif __name__ == \"__main__\":\n    main()\n```\n\nThe `try/except/else` structure guarantees exactly one push per run. Success and failure both produce a message. If the process is killed by the OOM killer or the machine reboots, you get nothing — and that absence is itself the signal. A missing daily report at 09:00 is a problem worth investigating.\n\nTwo refinements on top of this:\n\n**Heartbeat for critical jobs.** If a job must run daily and its absence is not obvious, add a separate dead-man's-switch. A tiny script that checks whether today's success push was recorded, and alerts if not. This catches the case where the job host is down entirely.\n\n**Deduplicate repeated failures.** A job that fails every five minutes will send 288 alerts a day. Add a simple state file: if the same failure signature fired in the last hour, suppress the push and log instead.\n\n```python\nimport hashlib\nimport json\nfrom pathlib import Path\n\nSTATE = Path(\"/var/lib/jobs/last_alert.json\")\n\n\ndef should_alert(signature: str, cooldown_s: int = 3600) -\u003e bool:\n    now = time.time()\n    try:\n        state = json.loads(STATE.read_text())\n    except (FileNotFoundError, json.JSONDecodeError):\n        state = {}\n\n    key = hashlib.sha1(signature.encode()).hexdigest()\n    last = state.get(key, 0)\n    if now - last \u003c cooldown_s:\n        return False\n\n    state[key] = now\n    STATE.write_text(json.dumps(state))\n    return True\n```\n\nCall `should_alert` before the failure push. The signature can be the exception type plus the failing function name — enough to group identical failures without collapsing distinct ones.\n\n## Operational notes\n\nA few things I learned the hard way:\n\n- **Token rotation.** If a token leaks, anyone can push to your WeChat. Rotate it from the console and update the environment variable. Because the token lives in an env var, rotation is a one-line change and a restart, not a code edit.\n- **Timezone.** PushPlus timestamps use the server's timezone. If your job runs in UTC and you read the push in local time, put the local date in the title explicitly, as the templates above do.\n- **Message size.** Very large markdown bodies get truncated or rejected. Cap content at a few thousand characters and link to a full report if you need more.\n- **Quota awareness.** Log the `data` field from successful responses — it contains the message ID, useful for debugging delivery. If you approach the daily quota, batch multiple metrics into one push rather than sending several.\n\nThe whole integration is maybe sixty lines of Python: a token in the environment, a send function with timeout and retry, a naming convention, two templates, and a hard rule that every job reports its own outcome. That is enough to make an unattended pipeline observable from your pocket.\n\nMore notes like this ship every week on this site.\n\n---\n\n## Daily Picks\n\n\u003e The following pairs are selected from the multi-timeframe trend scanner (Gate.io futures) and are for technical-analysis study only — not investment advice.\n\u003e Data updated: 2026-10-10 08:46:32\n\n### Long\n\n| Pair | Signal | Price | Take Profit | Stop Loss | R/R |\n|------|--------|-------|-------------|-----------|-----|\n| NEAR | 高质量标的做多 | $4.9102 | $5.0821 | $4.812 | 1:1.8 |\n\n### Short\n\n| Pair | Signal | Price | Take Profit | Stop Loss | R/R |\n|------|--------|-------|-------------|-----------|-----|\n| MAGIC | 热点跟涨 | $0.1247 | $0.1155 | $0.1291 | 1:2.1 |\n\n_2 picks selected. Scanner runs every 15 minutes._","positive_reactions_count":0,"cover_image":null,"tag_list":["pushplus","wechat","notifications","python"],"canonical_url":"https://www.hkcode.dpdns.org/en/posts/13.html","reading_time_minutes":7,"user":{"name":"mattleeee","username":"mattleeee","twitter_username":null,"github_username":"mattleeee","user_id":4160029,"website_url":null,"profile_image":"https://media2.dev.to/dynamic/image/width=640,height=640,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png","profile_image_90":"https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png"}},{"type_of":"article","id":4817125,"title":"GB28181 Camera Cascading: When SIP INVITE Never Arrived","description":"Heartbeats flowed, the platform showed the device online, but pressing play produced nothing. The SIP...","published":true,"published_at":"2026-10-08T10:04:30.665Z","slug":"gb28181-camera-cascading-when-sip-invite-never-arrived-3e1h","path":"/mattleeee/gb28181-camera-cascading-when-sip-invite-never-arrived-3e1h","url":"https://dev.to/mattleeee/gb28181-camera-cascading-when-sip-invite-never-arrived-3e1h","comments_count":1,"public_reactions_count":1,"page_views_count":94,"published_timestamp":"2026-10-08T10:04:30Z","body_markdown":"Heartbeats flowed, the platform showed the device online, but pressing play produced nothing. The SIP INVITE that should have kicked off the media session never reached the NCG. This is a walkthrough of how we localized that failure and what evidence we handed to the firewall engineer.\n\n## The Symptom\n\nA school's cameras cascade to a provincial education video platform through a Hikvision NCG (network control gateway). The platform client showed the device as online. Playback requests returned no video. No error dialog, no timeout message on the client side — just silence.\n\nThe first instinct is to blame the camera or the NCG. The second instinct is to blame the platform. Both are usually wrong, and in this case both were wrong.\n\n## GB28181 Call Flow: Two Different Worlds\n\nGB28181 sits on top of SIP, but the reliability characteristics of its two main message classes are very different. Understanding this split is what makes the diagnosis tractable.\n\n### Heartbeat: REGISTER and MESSAGE\n\nThe device keeps itself alive with periodic `REGISTER` and `MESSAGE` (keepalive) packets. These are small, sent at fixed intervals (typically 60s), and mostly stateful on the device side. If one is lost, the next one arrives within a minute and the platform never notices.\n\n```plaintext\nDevice                          Platform\n  |--- REGISTER (Expires: 3600) ---\u003e|\n  |\u003c-- 200 OK ----------------------|\n  |--- MESSAGE (Keepalive) --------\u003e|\n  |\u003c-- 200 OK ----------------------|\n  |            ...every 60s...\n```\n\nBecause these packets are small (a few hundred bytes) and periodic, they traverse NAT and firewall state tables easily. A NAT entry created by an outbound REGISTER stays alive because the keepalive refreshes it. From the platform's perspective, the device is online. From the network's perspective, a bidirectional path exists.\n\n### Media Session: INVITE / ACK / BYE\n\nA playback request is a different animal. The platform sends an `INVITE` to the device, the device answers with `200 OK` (or `486 Busy` etc.), the platform sends `ACK`, and then RTP flows. The INVITE is larger — the SDP body carries media negotiation — and it is **unsolicited from the device's point of view**. It arrives from the platform, not in response to anything the device just sent.\n\n```plaintext\nPlatform                        Device (NCG)\n  |--- INVITE (SDP, ~950B) ------\u003e|\n  |\u003c-- 100 Trying -----------------|\n  |\u003c-- 200 OK (SDP) ---------------|\n  |--- ACK -----------------------\u003e|\n  |\u003c========== RTP ==============\u003e|\n  |--- BYE -----------------------\u003e|\n```\n\nHere is the trap. If the device sits behind NAT and the firewall has no pinhole for inbound SIP, the INVITE is dropped before it ever reaches the NCG. The device has no idea a call was attempted. The platform sees its INVITE vanish. The client shows a spinner and then nothing.\n\nHeartbeats work because the device initiates them. INVITEs fail because the platform initiates them. Same SIP port, same session, completely different NAT direction.\n\n## Placing tcpdump Correctly\n\nThe most common mistake is running tcpdump on the wrong host. If you capture on the camera, you will see heartbeats and nothing else — because the INVITE never got that far. If you capture on the platform, you will see the platform sending INVITEs and no responses. Neither capture tells you **where** the packet died.\n\nCapture on the NCG server, the box that terminates the SIP signaling and sits between the school LAN and the WAN. That is the boundary. Anything that arrives there is inside the school. Anything that does not arrive there but was seen on the platform is stuck in the firewall or an upstream device.\n\n```bash\n# On the NCG server, capture SIP signaling both directions.\n# -i any so we don't guess the interface.\n# -s 0 so the SDP body is not truncated.\n# -w to a file so we can analyse offline.\nsudo tcpdump -i any -s 0 -w /tmp/sip_capture.pcap \\\n    'port 5060 or port 5061'\n```\n\nRun it for at least 3.5 minutes. Why 3.5? Because the platform's retry logic for INVITE typically fires at 500ms, 1s, 2s, 4s, 8s, 16s, 32s, 64s... and the client may retry the whole call. Three and a half minutes gives you two full retry ladders and enough heartbeat cycles to confirm the baseline is healthy. In our case, 16 retry bursts of roughly 950 bytes each were visible on the platform side. Zero arrived at the NCG.\n\n### Reading the Capture\n\nThe capture told a clean story. Heartbeats in both directions, REGISTER/200 OK pairs every 60 seconds, no gaps. Then, at the moment the operator pressed play on the client, nothing on the NCG side. No INVITE. No 100 Trying. No 950-byte burst.\n\n```bash\n# Count SIP methods seen at the NCG.\ntshark -r /tmp/sip_capture.pcap -Y sip \\\n    -T fields -e sip.Method | sort | uniq -c\n\n# Expected output on a healthy NCG:\n#   14 REGISTER\n#   14 200\n#   2  MESSAGE\n#   2  200\n#   0  INVITE   \u003c-- this is the smoking gun\n```\n\nA healthy capture would show at least one INVITE per playback attempt, plus the corresponding 100 Trying and 200 OK. Zero INVITEs at the NCG, while the platform confirms it sent them, means the packet was dropped upstream. The failure is not in the camera, not in the NCG, not in the platform. It is in the path.\n\n## Building the Evidence Pack\n\nFirewall engineers are busy. Do not send them a pcap and a paragraph. Send them a table.\n\n### 1. The Capture\n\nOne pcap from the NCG, one from the platform side if you can get it. Name them with timestamps and hostnames: `ncg01_20240315_1420.pcap`, `platform_edge_20240315_1420.pcap`. The platform-side capture is what proves the INVITE left the platform — without it, the firewall engineer can argue the platform never sent anything.\n\n### 2. The Five-Tuple Table\n\nExtract the exact tuples from both captures and diff them.\n\n```python\nimport subprocess\nimport csv\nfrom collections import Counter\n\ndef extract_sip_tuples(pcap_path):\n    \"\"\"Pull src/dst IP, ports, and SIP method from a pcap.\"\"\"\n    cmd = [\n        \"tshark\", \"-r\", pcap_path,\n        \"-Y\", \"sip\",\n        \"-T\", \"fields\",\n        \"-e\", \"ip.src\", \"-e\", \"ip.dst\",\n        \"-e\", \"udp.srcport\", \"-e\", \"udp.dstport\",\n        \"-e\", \"sip.Method\",\n        \"-e\", \"frame.time_epoch\",\n        \"-E\", \"separator=|\",\n    ]\n    out = subprocess.run(cmd, capture_output=True, text=True, check=True)\n    rows = []\n    for line in out.stdout.strip().splitlines():\n        parts = line.split(\"|\")\n        if len(parts) \u003e= 6:\n            rows.append({\n                \"src\": parts[0], \"dst\": parts[1],\n                \"sport\": parts[2], \"dport\": parts[3],\n                \"method\": parts[4], \"ts\": float(parts[5]),\n            })\n    return rows\n\nncg = extract_sip_tuples(\"/tmp/ncg01.pcap\")\nplat = extract_sip_tuples(\"/tmp/platform_edge.pcap\")\n\ndef tuple_key(r):\n    return (r[\"src\"], r[\"dst\"], r[\"sport\"], r[\"dport\"])\n\nncg_keys = Counter(tuple_key(r) for r in ncg)\nplat_keys = Counter(tuple_key(r) for r in plat)\n\nprint(\"Tuples seen on platform but NOT on NCG:\")\nfor k, count in plat_keys.items():\n    if k not in ncg_keys:\n        print(f\"  {k}  x{count}\")\n```\n\nFor our case, the output was a single line: the platform's SIP signaling IP and port, destined for the NCG's public IP and port 5060, with roughly 16 occurrences. That is the exact tuple the firewall needs to permit.\n\n### 3. The Timeline\n\nA short timeline matters more than raw packet dumps. Something like:\n\n```plaintext\n14:22:01  REGISTER from NCG -\u003e platform, 200 OK. Baseline healthy.\n14:25:33  Operator presses play on platform client.\n14:25:33  Platform sends INVITE to NCG public IP:5060 (seen on platform capture).\n14:25:33  No INVITE arrives at NCG. No 100 Trying, no 200 OK.\n14:25:34  Platform retries INVITE. Same result.\n14:25:36  Platform retries. Same result.\n...\n14:26:37  Final retry. Operator gives up.\n```\n\nThis tells the firewall engineer exactly what to look for in their logs: dropped packets on the WAN interface, destination port 5060, source IP of the platform, around 14:25:33.\n\n## The Actual Fix\n\nIn our case, the school firewall had a stateful rule permitting outbound SIP from the NCG to the platform, but no corresponding inbound rule for the platform's SIP signaling IP. The outbound REGISTER created a NAT pinhole, but the pinhole was UDP and tied to the source port the NCG used. The platform's INVITE came from a different source port on its side, so it did not match the pinhole and was dropped.\n\nThe fix was a static inbound rule: permit UDP from the platform's SIP signaling IP (a small CIDR, not a single host — platforms often have a signaling cluster) to the NCG's public IP on port 5060. RTP ports 30000-30500 were opened in the same rule set. After that, INVITEs arrived, 200 OKs went back, and playback worked.\n\nA note on why this is easy to miss: the platform showed the device as online because heartbeats were flowing. That status is derived from REGISTER state, not from INVITE reachability. Online in the platform UI does not mean the call path is open. Treat them as two separate reachability questions.\n\n## A Reusable Check\n\nBefore you escalate to the firewall team, run this on the NCG to confirm the split between outbound-initiated and inbound-initiated traffic:\n\n```python\nimport subprocess\n\ndef sip_method_counts(pcap_path):\n    \"\"\"Return a dict of SIP method -\u003e count.\"\"\"\n    cmd = [\n        \"tshark\", \"-r\", pcap_path,\n        \"-Y\", \"sip\",\n        \"-T\", \"fields\", \"-e\", \"sip.Method\",\n    ]\n    out = subprocess.run(cmd, capture_output=True, text=True, check=True)\n    counts = {}\n    for m in out.stdout.strip().splitlines():\n        counts[m] = counts.get(m, 0) + 1\n    return counts\n\ncounts = sip_method_counts(\"/tmp/ncg01.pcap\")\nprint(counts)\n\nif counts.get(\"REGISTER\", 0) \u003e 0 and counts.get(\"INVITE\", 0) == 0:\n    print(\"Outbound path OK, inbound path blocked. Escalate to firewall.\")\nelif counts.get(\"INVITE\", 0) \u003e 0 and counts.get(\"200\", 0) == 0:\n    print(\"INVITE arrives but device does not answer. Check NCG/camera.\")\nelse:\n    print(\"Both paths appear active. Look at RTP or SDP.\")\n```\n\nThe heuristic is simple: if REGISTER is present but INVITE is absent, the problem is directional. Outbound works, inbound does not. That is almost always a firewall or NAT rule, not a device fault.\n\n## Takeaways\n\n- Heartbeats prove outbound reachability, not inbound. They are independent questions.\n- Capture at the boundary (the NCG), not at the endpoints, to localize where the packet dies.\n- Run the capture long enough to cover the platform's retry ladder — 3.5 minutes is a safe floor.\n- Hand the firewall engineer a five-tuple table and a timeline, not a pcap and a hunch.\n- The fix is usually a static inbound rule for the platform's SIP signaling IP and RTP port range.\n\nMore notes like this ship every week on this site.\n\n---\n\n## Daily Picks\n\n\u003e The following pairs are selected from the multi-timeframe trend scanner (Gate.io futures) and are for technical-analysis study only — not investment advice.\n\u003e Data updated: 2026-10-08 08:46:21\n\n### Short\n\n| Pair | Signal | Price | Take Profit | Stop Loss | R/R |\n|------|--------|-------|-------------|-----------|-----|\n| SUE | 弱做空 | $0.0006 | $0.0006 | $0.0006 | 1:1.2 |\n\n_1 picks selected. Scanner runs every 15 minutes._","positive_reactions_count":1,"cover_image":null,"tag_list":["gb28181","sip","invite","tcpdump"],"canonical_url":"https://www.hkcode.dpdns.org/en/posts/11.html","reading_time_minutes":7,"user":{"name":"mattleeee","username":"mattleeee","twitter_username":null,"github_username":"mattleeee","user_id":4160029,"website_url":null,"profile_image":"https://media2.dev.to/dynamic/image/width=640,height=640,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png","profile_image_90":"https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png"}},{"type_of":"article","id":4804506,"title":"Surviving Corporate HTTPS Interception: Life Behind Sangfor aTrust","description":"Corporate laptops are strange machines. You have admin rights on paper, but every outbound TLS...","published":true,"published_at":"2026-10-06T04:54:16.863Z","slug":"surviving-corporate-https-interception-life-behind-sangfor-atrust-2i42","path":"/mattleeee/surviving-corporate-https-interception-life-behind-sangfor-atrust-2i42","url":"https://dev.to/mattleeee/surviving-corporate-https-interception-life-behind-sangfor-atrust-2i42","comments_count":0,"public_reactions_count":0,"page_views_count":61,"published_timestamp":"2026-10-06T04:54:16Z","body_markdown":"Corporate laptops are strange machines. You have admin rights on paper, but every outbound TLS connection is terminated and re-originated by a local agent. Sangfor aTrust is one of the more common zero-trust clients in APAC enterprises, and it does exactly this: it installs a root CA into the OS trust store, proxies traffic through a local service, and re-signs certificates on the fly. Your browser shows a green padlock. `pip install` does not.\n\nThis post is a field guide to living with that setup without getting your laptop quarantined.\n\n## How the interception actually works\n\naTrust runs a local proxy (typically on `127.0.0.1` on some port) and pushes a PAC file or system proxy setting. When you connect to `pypi.org`, the agent completes the TLS handshake with the real server using its own credentials, then presents *you* a certificate signed by the corporate root CA. The OS trust store contains that CA, so anything using the system store is fine. Anything shipping its own bundle is not.\n\nThe tell is always in the chain. Run:\n\n```bash\nopenssl s_client -connect pypi.org:443 -servername pypi.org -showcerts \u003c/dev/null 2\u003e/dev/null \\\n  | openssl x509 -noout -issuer -subject\n```\n\nOn a clean network you get something like `issuer=C=US, O=Let's Encrypt, CN=R3`. Behind aTrust you get `issuer=C=CN, O=Sangfor Technologies, CN=Sangfor SSL Proxy CA` or an internal enterprise CA name. That single line is the difference between a working environment and a day of debugging.\n\n### Distinguishing MITM from a normal corporate proxy\n\nA plain forward proxy does not terminate TLS — it just tunnels bytes and the issuer stays untouched. MITM means the issuer changed. To confirm it's the agent and not a rogue AP:\n\n1. Compare the issuer seen by `openssl s_client` against the issuer shown in the browser's certificate viewer for the same host. If they match and both are internal, it's the agent.\n2. Check the local proxy port: `netstat -ano | findstr LISTENING` on Windows, then match the PID to the aTrust process.\n3. Hit a host with a known public chain, e.g. `openssl s_client -connect github.com:443`. If GitHub is also re-signed, everything is being intercepted.\n\nOnce you know the CA's subject and fingerprint, you can decide per-tool what to do.\n\n## Which tools tolerate the corporate CA\n\nThe dividing line is *where the trust anchor comes from*.\n\n**Tolerates it (uses OS trust store):**\n- Chrome, Edge, Firefox (with `security.enterprise_roots.enabled=true`)\n- Windows-native HTTP stacks, .NET, WinHTTP\n- `curl` on Windows when built against Schannel\n- macOS `curl` and `NSURLSession`-based tools\n- Python `requests` *only if* you point it at the system store or a bundle containing the corporate CA\n\n**Does not tolerate it (ships its own bundle or pins):**\n- Python `certifi` (bundled, frozen at release time)\n- Node.js (bundles Mozilla's CA set)\n- Git for Windows (ships its own `ca-bundle.crt`)\n- Rust `reqwest` default roots\n- Anything doing certificate pinning: mobile SDKs, some CLI tools, `go` binaries with embedded roots\n\nThe fix is almost always the same: build a bundle that contains both the Mozilla roots and the corporate root, then point the tool at it.\n\n### Exporting the corporate root CA\n\nOn Windows:\n\n```powershell\n# List candidate roots, then export the one you identified\nGet-ChildItem Cert:\\LocalMachine\\Root | Where-Object { $_.Subject -like \"*Sangfor*\" } |\n  ForEach-Object { Export-Certificate -Cert $_ -FilePath \"$env:USERPROFILE\\corp-root.cer\" }\n```\n\nThen convert to PEM. If you have OpenSSL or Git's bundled one:\n\n```bash\nopenssl x509 -inform DER -in corp-root.cer -out corp-root.pem\n```\n\nKeep this file somewhere stable, e.g. `~/.certs/corp-root.pem`.\n\n## Fixing Python: certifi, requests, and pip\n\n`certifi` is a vendored snapshot. The clean approach is to append the corporate root to a *copy* and set `REQUESTS_CA_BUNDLE` / `SSL_CERT_FILE`, not to mutate the installed certifi package (which breaks on every upgrade).\n\n```python\nimport os\nimport shutil\nfrom pathlib import Path\n\nimport certifi\n\nCORP_ROOT = Path.home() / \".certs\" / \"corp-root.pem\"\nBUNDLE = Path.home() / \".certs\" / \"combined-ca.pem\"\n\n\ndef build_combined_bundle() -\u003e Path:\n    \"\"\"Concatenate certifi's roots with the corporate root CA.\"\"\"\n    if not CORP_ROOT.exists():\n        raise FileNotFoundError(f\"missing corporate root: {CORP_ROOT}\")\n\n    with BUNDLE.open(\"wb\") as out:\n        out.write(Path(certifi.where()).read_bytes())\n        out.write(b\"\\n\")\n        out.write(CORP_ROOT.read_bytes())\n\n    return BUNDLE\n\n\nif __name__ == \"__main__\":\n    bundle = build_combined_bundle()\n    os.environ[\"REQUESTS_CA_BUNDLE\"] = str(bundle)\n    os.environ[\"SSL_CERT_FILE\"] = str(bundle)\n\n    import requests\n\n    r = requests.get(\"https://pypi.org/simple/\", timeout=10)\n    print(r.status_code, len(r.content))\n```\n\nSet the two environment variables globally (`REQUESTS_CA_BUNDLE` for `requests`, `SSL_CERT_FILE` for the stdlib `ssl` module and most other tools) and most Python breakage disappears. For `pip` specifically, either rely on `SSL_CERT_FILE` or set it explicitly:\n\n```bash\npip config set global.cert ~/.certs/combined-ca.pem\n```\n\nIf a package still fails, it's probably pinning. Check with:\n\n```python\nimport ssl, socket\n\nhost = \"pypi.org\"\nctx = ssl.create_default_context(cafile=str(BUNDLE))\nwith socket.create_connection((host, 443), timeout=10) as sock:\n    with ctx.wrap_socket(sock, server_hostname=host) as ssock:\n        print(ssock.getpeercert()[\"issuer\"])\n```\n\nIf that succeeds but the package fails, the package is doing its own pinning or bundling — you'll need its own override (e.g. `NODE_EXTRA_CA_CERTS` for Node, `SSL_CERT_DIR` for Go).\n\n## Fixing Git\n\nGit for Windows ignores the Windows store by default. Two options: point it at the combined bundle, or tell it to use Schannel.\n\nOption A — explicit bundle:\n\n```bash\ngit config --global http.sslCAInfo ~/.certs/combined-ca.pem\ngit config --global http.sslBackend openssl\n```\n\nOption B — use the OS store (Windows only, often the cleanest):\n\n```bash\ngit config --global http.sslBackend schannel\n```\n\nWith Schannel, Git trusts whatever Windows trusts, including the aTrust root. This is my default on corporate Windows machines. On macOS, `http.sslBackend` doesn't exist; use the combined bundle approach there.\n\nVerify:\n\n```bash\nGIT_CURL_VERBOSE=1 git ls-remote https://github.com/git/git HEAD 2\u003e\u00261 | grep -i \"issuer\\|SSL certificate\"\n```\n\nIf you see the corporate issuer in the verbose output and the command succeeds, you're done.\n\n### SSH is usually untouched\n\naTrust typically proxies HTTP/HTTPS. SSH over port 22 or 443 is usually passed through or blocked outright, not MITM'd. If `git clone [email protected]:...` works, prefer it — SSH keys don't care about CA chains. If port 22 is blocked, GitHub's `ssh.github.com:443` endpoint is the standard workaround:\n\n```ssh\nHost github.com\n  HostName ssh.github.com\n  Port 443\n  User git\n```\n\n## Routing around blocked hosts, politely\n\nSome hosts are blocked by policy, not by the proxy's technical limits. The wrong move is to spin up a personal VPN and tunnel everything — that's the fastest way to a security incident. The right move depends on the reason for the block.\n\n**If it's a category block (e.g. a package registry mirrors the security team hasn't approved):** file a ticket. Internal mirrors of PyPI, npm, and Go modules are common in these environments, and using them is both faster and sanctioned. Ask for the internal index URL; it's usually already configured for some team.\n\n**If it's an allowlist model:** the block is on the destination, and the proxy will happily pass traffic to approved hosts. Route around it by using an approved equivalent. For container images, that often means the company's Harbor or Artifactory instance rather than Docker Hub.\n\n**If you need to reach a host for legitimate work and it's genuinely missing from the allowlist:** the polite ask is specific. \"I need `registry.npmjs.org` for build reproducibility on project X; the internal mirror at `npm.internal.corp` is missing package Y as of version Z.\" Concrete, scoped, and easy for a security engineer to approve.\n\nWhat you should *not* do: install a third-party VPN, use a public SOCKS proxy, or disable the aTrust agent. All three are detectable and all three turn a five-minute ticket into a meeting with your manager.\n\n### A note on certificate pinning\n\nSome internal tools pin their own certs and will break the moment the corporate CA changes. If you maintain such a tool, don't pin the leaf; pin the corporate root's public key and make it configurable. A hardcoded SPKI hash is a time bomb.\n\n```python\nimport hashlib\nimport ssl\n\ndef spki_sha256(host: str, port: int = 443) -\u003e str:\n    \"\"\"Return the base64 SHA-256 of the leaf's SubjectPublicKeyInfo.\"\"\"\n    der = ssl.get_server_certificate((host, port), ca_certs=str(BUNDLE)).encode()\n    # In practice, parse with cryptography.x509 and hash the SPKI, not the whole cert.\n    return hashlib.sha256(der).hexdigest()\n```\n\nThe point isn't the exact hash — it's that pinning should be a config value, not a constant buried in source.\n\n## A short checklist\n\nWhen something breaks behind aTrust:\n\n1. Confirm the issuer with `openssl s_client`. If it's the corporate CA, you know the cause.\n2. Build `~/.certs/combined-ca.pem` from `certifi` plus the corporate root.\n3. Export `SSL_CERT_FILE` and `REQUESTS_CA_BUNDLE` to that path.\n4. For Git, set `http.sslBackend=schannel` on Windows or `http.sslCAInfo` everywhere.\n5. Prefer SSH where possible; it sidesteps the whole problem.\n6. For blocked hosts, use internal mirrors or file a scoped ticket — don't tunnel.\n\nThe interception isn't going away. Once you know where the trust anchors come from, most of the friction disappears.\n\nMore notes like this ship every week on this site.\n\n---\n\n## Daily Picks\n\n\u003e The following pairs are selected from the multi-timeframe trend scanner (Gate.io futures) and are for technical-analysis study only — not investment advice.\n\u003e Data updated: 2026-10-06 12:36:33\n\n### Long\n\n| Pair | Signal | Price | Take Profit | Stop Loss | R/R |\n|------|--------|-------|-------------|-----------|-----|\n| SKYAI |  | $0.0416 | $0.0433 | $0.0406 | 1:1.6 |\n| RE |  | $0.4991 | $0.5181 | $0.4866 | 1:1.5 |\n\n_2 picks selected. Scanner runs every 15 minutes._\n","positive_reactions_count":0,"cover_image":null,"tag_list":["sangforatrust","mitm","tls","proxy"],"canonical_url":"https://www.hkcode.dpdns.org/en/posts/9.html","reading_time_minutes":6,"user":{"name":"mattleeee","username":"mattleeee","twitter_username":null,"github_username":"mattleeee","user_id":4160029,"website_url":null,"profile_image":"https://media2.dev.to/dynamic/image/width=640,height=640,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png","profile_image_90":"https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png"}},{"type_of":"article","id":4804481,"title":"Health Checks for a Fleet of Scheduled Tasks","description":"Nine unattended scheduled tasks is a fleet, whether you call it that or not. Once you pass three or...","published":true,"published_at":"2026-10-06T04:44:45.429Z","slug":"health-checks-for-a-fleet-of-scheduled-tasks-5db7","path":"/mattleeee/health-checks-for-a-fleet-of-scheduled-tasks-5db7","url":"https://dev.to/mattleeee/health-checks-for-a-fleet-of-scheduled-tasks-5db7","comments_count":0,"public_reactions_count":0,"page_views_count":20,"published_timestamp":"2026-10-06T04:44:45Z","body_markdown":"Nine unattended scheduled tasks is a fleet, whether you call it that or not. Once you pass three or four, checking `taskschd.msc` by hand stops being a habit and starts being a liability — you only look when something is visibly broken, which is exactly the failure mode you were trying to avoid by scheduling the work in the first place.\n\nThis post covers a small audit script that enumerates a set of tasks by name pattern, reads their state and last result, compares them against an expected schedule, and pushes an alert when something is wrong. It runs as a scheduled task itself, because a monitoring script that depends on you remembering to run it is not monitoring.\n\n## What the Task Scheduler actually exposes\n\nThe Task Scheduler COM API (`Schedule.Service`) is the authoritative source. The `schtasks.exe` text output is fine for humans and terrible for parsing — column widths shift, and localized Windows installs translate the field names. Go through COM.\n\n```python\nimport win32com.client\n\nTASK_ENUM_HIDDEN = 1\n\ndef connect():\n    service = win32com.client.Dispatch(\"Schedule.Service\")\n    service.Connect()\n    return service\n\ndef iter_tasks(service, folder_path=\"\\\\\"):\n    folder = service.GetFolder(folder_path)\n    for task in folder.GetTasks(TASK_ENUM_HIDDEN):\n        yield task\n```\n\nEach `IRegisteredTask` gives you a handful of fields worth reading:\n\n| Field | Type | What it tells you |\n|---|---|---|\n| `Name` | str | Task name, without folder path |\n| `Path` | str | Full path including folder, e.g. `\\Quant\\Quant_Factor` |\n| `State` | int | `0` Unknown, `1` Disabled, `2` Queued, `3` Ready, `4` Running |\n| `Enabled` | bool | Separate from state; a disabled task can still report `Ready` in some edge cases |\n| `LastRunTime` | datetime | When the last run *started* |\n| `LastTaskResult` | int | Exit code of the last completed run |\n| `NextRunTime` | datetime | Next scheduled fire, `1899-12-30` if none |\n\nTwo traps here. First, `State` and `Enabled` are not redundant. A task can be enabled but sitting in `Disabled` state if its trigger was disabled, or disabled at the task level while the state enum lags. Check both.\n\nSecond, `LastTaskResult` is only meaningful after a run *completes*. While a task is running, `LastTaskResult` still holds the previous run's value. If you alert on a non-zero result without checking `State == 4`, you will page yourself about a task that is mid-run and about to succeed.\n\nThe `1899-12-30` sentinel for \"no next run\" is a COM date artifact — it's the zero of the OLE automation date format. Treat anything before 1980 as \"none\".\n\n## Defining what \"healthy\" means\n\nThe script needs a table of expectations, because \"task exists and ran once\" is not a health check. Each task has a cadence, and a task that last ran three days ago when it should run hourly is broken even if its exit code was zero.\n\n```python\nfrom dataclasses import dataclass\nfrom datetime import timedelta\n\n@dataclass(frozen=True)\nclass Expectation:\n    pattern: str            # fnmatch pattern against task Path\n    max_age: timedelta      # how stale LastRunTime may be\n    allow_disabled: bool = False\n    description: str = \"\"\n\nEXPECTED = [\n    Expectation(\"\\\\Quant\\\\Quant_*\",        timedelta(hours=2),  description=\"quant pipeline\"),\n    Expectation(\"\\\\TeleAgent_*\",           timedelta(minutes=30), description=\"telegram agent\"),\n    Expectation(\"\\\\Quant\\\\Quant_Nightly\",  timedelta(hours=26), description=\"nightly batch\"),\n]\n```\n\n`max_age` should be generous enough to absorb a single missed cycle plus jitter, but not so generous that a task can silently skip twice. For an hourly task, two hours is a reasonable ceiling. For a nightly job, 26 hours covers a late start without hiding a full day of silence.\n\nThe `allow_disabled` flag matters for tasks you intentionally park. Without it, every disabled task shows up as a finding, and you learn to ignore the alerts — which defeats the point.\n\n## Classifying each task\n\nThe classification logic is where most naive audits go wrong. There are four distinct failure modes and they need different messages.\n\n```python\nimport fnmatch\nfrom datetime import datetime, timezone\n\nSTATE_UNKNOWN, STATE_DISABLED, STATE_QUEUED, STATE_READY, STATE_RUNNING = range(5)\n\ndef classify(task, expectation, now):\n    \"\"\"Return (severity, reason) where severity is 'ok', 'warn' or 'fail'.\"\"\"\n    state = task.State\n    enabled = task.Enabled\n    last_result = task.LastTaskResult\n    last_run = task.LastRunTime\n\n    if not enabled or state == STATE_DISABLED:\n        if expectation.allow_disabled:\n            return \"ok\", \"disabled by design\"\n        return \"fail\", \"task is disabled\"\n\n    if state == STATE_RUNNING:\n        # Don't judge LastTaskResult while a run is in flight.\n        return \"ok\", \"currently running\"\n\n    if last_result != 0:\n        return \"fail\", f\"last result {last_result:#010x}\"\n\n    if last_run is None or last_run.year \u003c 1980:\n        return \"fail\", \"never ran\"\n\n    # COM returns naive local datetimes; compare in local time.\n    age = now - last_run\n    if age \u003e expectation.max_age:\n        hours = age.total_seconds() / 3600\n        return \"fail\", f\"stale, last run {hours:.1f}h ago\"\n\n    return \"ok\", f\"ok, last run {age.total_seconds()/60:.0f}m ago\"\n```\n\nThe `last_result` non-zero check catches the common case: the task fired, the script inside exited with code 1, and nobody noticed because the scheduler doesn't care. `0x80070002` (file not found) and `0x80070005` (access denied) are the two you will see most often when someone moves a script or changes a service account.\n\nNote the running-state short-circuit. A long-running task that takes 90 minutes will otherwise trip the staleness check on every audit pass, because `LastRunTime` only updates when a run *starts*. If your task runs longer than your `max_age`, you need to either raise `max_age` or check `State == 4` before evaluating staleness — which is what the code above does.\n\n## Walking the folder tree\n\nTasks live in folders, and `GetTasks` only returns the tasks directly in the folder you asked for. If your tasks are organized under `\\Quant\\` and `\\`, you need to recurse.\n\n```python\ndef walk(service, folder_path=\"\\\\\"):\n    folder = service.GetFolder(folder_path)\n    for task in folder.GetTasks(TASK_ENUM_HIDDEN):\n        yield task\n    for sub in folder.GetFolders(0):\n        yield from walk(service, sub.Path)\n```\n\n`GetFolders(0)` returns immediate subfolders only; the recursion handles the rest. The `0` flag is a reserved bitmask that must be zero — passing anything else raises.\n\n## Matching tasks to expectations\n\nUse `fnmatch` on the full `Path`, not the bare `Name`. Two folders can hold tasks with the same name, and matching on `Name` alone will silently pick the wrong one.\n\n```python\ndef audit(service, expectations, now):\n    findings = []\n    seen = set()\n\n    for task in walk(service):\n        for exp in expectations:\n            if fnmatch.fnmatch(task.Path, exp.pattern):\n                severity, reason = classify(task, exp, now)\n                findings.append({\n                    \"path\": task.Path,\n                    \"severity\": severity,\n                    \"reason\": reason,\n                    \"result\": task.LastTaskResult,\n                    \"last_run\": task.LastRunTime,\n                })\n                seen.add(exp.pattern)\n                break\n\n    # Expectations that matched nothing are themselves a finding.\n    for exp in expectations:\n        if exp.pattern not in seen:\n            findings.append({\n                \"path\": exp.pattern,\n                \"severity\": \"fail\",\n                \"reason\": \"no matching task found\",\n                \"result\": None,\n                \"last_run\": None,\n            })\n\n    return findings\n```\n\nThe unmatched-expectation check is the one people skip, and it's the most valuable. A task that was deleted, renamed, or moved to a different folder disappears from the enumeration entirely — a naive audit reports \"all clear\" because there is nothing to complain about. The expectation table is what makes absence detectable.\n\n## The alert\n\nPushPlus takes a token and a message body over HTTP POST and delivers to WeChat. Keep the alert short: a subject line that says how many tasks failed, and a body that lists only the failures. Nobody reads a wall of green checkmarks.\n\n```python\nimport requests\n\nPUSHPLUS_TOKEN = \"your-token-here\"\n\ndef format_alert(findings):\n    bad = [f for f in findings if f[\"severity\"] != \"ok\"]\n    if not bad:\n        return None\n    lines = [f\"{len(bad)} task(s) need attention\", \"\"]\n    for f in bad:\n        lines.append(f\"{f['path']}\")\n        lines.append(f\"  {f['reason']}\")\n    return \"\\n\".join(lines)\n\ndef push(message):\n    resp = requests.post(\n        \"https://www.pushplus.plus/send\",\n        json={\n            \"token\": PUSHPLUS_TOKEN,\n            \"title\": \"Task fleet alert\",\n            \"content\": message,\n            \"template\": \"txt\",\n        },\n        timeout=15,\n    )\n    resp.raise_for_status()\n```\n\nSend only when there is something to say. An alert channel that fires every hour with \"all fine\" trains you to swipe past it, and then the one real alert gets swiped too.\n\n## Wiring it together\n\n```python\ndef main():\n    service = connect()\n    now = datetime.now()\n    findings = audit(service, EXPECTED, now)\n\n    for f in findings:\n        print(f\"{f['severity']:4} {f['path']:40} {f['reason']}\")\n\n    message = format_alert(findings)\n    if message:\n        push(message)\n\nif __name__ == \"__main__\":\n    main()\n```\n\nRun it manually once, fix whatever it reports, then register it as a scheduled task. A 15-minute cadence is fine — the audit is cheap, and a shorter interval means you catch a failure closer to when it happened.\n\n## Why the audit must itself be scheduled\n\nThe obvious objection: \"I'll just run this when I think about it.\" That is the same failure mode the audit exists to catch. If you remember to check, you don't need the check; if you forget, the check never runs. The only configuration where the audit has value is one where it runs without you.\n\nThat means the audit task is itself in the fleet, and it needs to be in the expectation table — a task that monitors nine tasks and is monitored by nobody is a single point of failure. Point an expectation at it with a `max_age` slightly larger than its own interval:\n\n```python\nExpectation(\"\\\\Quant\\\\Audit_TaskFleet\", timedelta(minutes=45), description=\"this audit\"),\n```\n\nIf the audit task dies, you stop receiving alerts — and the way you find out is that you stop receiving alerts, which is exactly the signal you would miss. There is no fully self-healing answer here, but you can reduce the blast radius: have the audit write a heartbeat row to a table or a file, and check that heartbeat from a second, independent mechanism (a cron job on a different host, a cloud function, anything that doesn't share the same scheduler). Two schedulers watching each other is more resilient than one scheduler watching itself.\n\n## What to watch over time\n\nThree signals are worth tracking beyond pass/fail:\n\n- **Run duration drift.** A task that used to finish in 40 seconds and now takes 12 minutes is heading for a timeout. Log `LastRunTime` deltas and alert on a sustained increase, not a single spike.\n- **Result code distribution.** A task that alternates between `0` and `0x80070002` is flapping. Count non-zero results over a rolling window rather than checking only the most recent.\n- **Stale-but-zero.** The nastiest case: exit code `0`, last run within tolerance, but the task is doing nothing because its input file is empty or its upstream dependency silently stopped producing. The scheduler can't see inside the process. If it matters, have the task itself emit a completion marker with a row count or a timestamp, and audit that marker instead of trusting the exit code.\n\nThe audit script is the first layer. It catches the failures the scheduler can see. The second layer — output validation — catches the ones it can't, and it belongs inside the task, not in the audit.\n\nMore notes like this ship every week on this site.\n\n---\n\n## Daily Picks\n\n\u003e The following pairs are selected from the multi-timeframe trend scanner (Gate.io futures) and are for technical-analysis study only — not investment advice.\n\u003e Data updated: 2026-10-06 12:36:33\n\n### Long\n\n| Pair | Signal | Price | Take Profit | Stop Loss | R/R |\n|------|--------|-------|-------------|-----------|-----|\n| SKYAI |  | $0.0416 | $0.0433 | $0.0406 | 1:1.6 |\n| RE |  | $0.4991 | $0.5181 | $0.4866 | 1:1.5 |\n\n_2 picks selected. Scanner runs every 15 minutes._\n","positive_reactions_count":0,"cover_image":null,"tag_list":["healthcheck","audit","lasttaskresult","scheduledtasks"],"canonical_url":"https://www.hkcode.dpdns.org/en/posts/7.html","reading_time_minutes":7,"user":{"name":"mattleeee","username":"mattleeee","twitter_username":null,"github_username":"mattleeee","user_id":4160029,"website_url":null,"profile_image":"https://media2.dev.to/dynamic/image/width=640,height=640,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png","profile_image_90":"https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png"}},{"type_of":"article","id":4804480,"title":"Migrating a Python Automation Stack to a New Windows Server","description":"Migrating a Python automation stack between Windows machines looks trivial until it isn't. Nine...","published":true,"published_at":"2026-10-06T04:44:07.795Z","slug":"migrating-a-python-automation-stack-to-a-new-windows-server-2b7p","path":"/mattleeee/migrating-a-python-automation-stack-to-a-new-windows-server-2b7p","url":"https://dev.to/mattleeee/migrating-a-python-automation-stack-to-a-new-windows-server-2b7p","comments_count":0,"public_reactions_count":0,"page_views_count":80,"published_timestamp":"2026-10-06T04:44:07Z","body_markdown":"Migrating a Python automation stack between Windows machines looks trivial until it isn't. Nine scheduled tasks, a blog pipeline, a trading system, and a few hundred state files moved from a Windows 11 dev box (UTF-8 default) to a fresh Windows Server 2016 instance (GBK default). The scripts were identical. The behavior was not. Here is the package approach that worked, including the encoding trap that cost the most time.\n\n## Why \"just copy the folder\" fails\n\nThe dev box had accumulated state that should not travel: logs, built site output, trade records, session cookies, and hundreds of JSON state files that referenced absolute paths from the old machine. Copying everything would have dragged stale data onto the server, and in the trading system's case, replayed old positions into a fresh account.\n\nThe other problem is that scheduled tasks store absolute paths. A task registered on `C:\\Users\\dev\\...` will not resolve on a server where the user profile is `C:\\Users\\Administrator\\...`. Re-registering tasks by hand across nine entries is a recipe for typos, so the migration needed to be scripted end to end.\n\n## The package layout\n\nI built a single `migration/` folder that could be copied to the new machine and run without editing:\n\n```plaintext\nmigration/\n  payload/            # the actual code, minus runtime data\n  exclude.txt         # xcopy exclusion list\n  install.bat         # one-click installer\n  register_tasks.ps1  # re-registers all scheduled tasks\n  verify.bat          # checks every task's action and last result\n  requirements.txt\n```\n\nThe `payload/` directory is the source tree with runtime directories stripped out. Everything machine-specific is resolved at install time.\n\n## xcopy with an exclusion list\n\n`robocopy` is the better tool for large trees, but `xcopy` with `/EXCLUDE` is simpler to reason about when you want a flat file list and predictable behavior on older Windows Server builds. The exclude file protects runtime data:\n\n```plaintext\n\\logs\\\n\\site\\build\\\n\\data\\trades\\\n\\data\\cookies\\\n\\state\\*.tmp\n\\__pycache__\\\n*.pyc\n.env\n```\n\nThen the copy is one line:\n\n```bat\nxcopy \"%~dp0payload\" \"%USERPROFILE%\\automation\" /E /I /Y /EXCLUDE:%~dp0exclude.txt\n```\n\n`/E` copies empty directories, `/I` assumes the destination is a directory if it doesn't exist, `/Y` suppresses the overwrite prompt. The exclude file uses substring matching, so `\\logs\\` catches any path segment named `logs` anywhere in the tree.\n\nOne caveat: xcopy's exclude matching is greedy and case-insensitive on Windows. If you have a source directory named `Logs` and a legitimate `catalog` directory, the pattern `\\logs\\` will not match `catalog` because it requires the surrounding backslashes, but a bare `logs` pattern would. Keep the delimiters.\n\n## One-click installer that auto-detects paths\n\nThe installer resolves everything relative to `%USERPROFILE%` so the same batch file works on any account:\n\n```bat\n@echo off\nsetlocal\n\nset \"ROOT=%USERPROFILE%\\automation\"\nset \"PY=%LOCALAPPDATA%\\Programs\\Python\\Python311\\python.exe\"\n\nif not exist \"%PY%\" (\n    echo Python not found at %PY%\n    exit /b 1\n)\n\necho Installing to %ROOT%\nxcopy \"%~dp0payload\" \"%ROOT%\" /E /I /Y /EXCLUDE:%~dp0exclude.txt\n\nif errorlevel 1 (\n    echo Copy failed\n    exit /b 1\n)\n\n\"%PY%\" -m pip install -r \"%~dp0requirements.txt\" --quiet\nif errorlevel 1 (\n    echo Dependency install failed\n    exit /b 1\n)\n\npowershell -ExecutionPolicy Bypass -File \"%~dp0register_tasks.ps1\" -Root \"%ROOT%\"\nif errorlevel 1 (\n    echo Task registration failed\n    exit /b 1\n)\n\necho Install complete\nendlocal\n```\n\nThe `%~dp0` prefix resolves to the directory containing the batch file, so the package can live anywhere. The installer fails fast at each stage instead of continuing with a broken environment.\n\n## Re-registering scheduled tasks\n\nNine tasks needed to move from the old machine to the new one. Rather than export XML and rewrite paths, I defined the tasks in a PowerShell array and used `New-ScheduledTaskAction` plus `Register-ScheduledTask`. This keeps the task definitions in source control and makes the paths parametric:\n\n```powershell\nparam(\n    [Parameter(Mandatory=$true)]\n    [string]$Root\n)\n\n$python = Join-Path $env:LOCALAPPDATA \"Programs\\Python\\Python311\\python.exe\"\n\n$tasks = @(\n    @{ Name = \"BlogPublish\";   Script = \"blog\\publish.py\";    Time = \"06:00\" },\n    @{ Name = \"BlogFetch\";     Script = \"blog\\fetch.py\";      Time = \"06:30\" },\n    @{ Name = \"TradeScan\";     Script = \"trading\\scan.py\";    Time = \"09:00\" },\n    @{ Name = \"TradeExecute\";  Script = \"trading\\execute.py\"; Time = \"09:15\" },\n    @{ Name = \"TradeReport\";   Script = \"trading\\report.py\";  Time = \"16:00\" },\n    @{ Name = \"StateBackup\";   Script = \"ops\\backup.py\";      Time = \"23:00\" },\n    @{ Name = \"HealthCheck\";   Script = \"ops\\health.py\";      Time = \"00:00\" },\n    @{ Name = \"LogRotate\";     Script = \"ops\\rotate.py\";      Time = \"00:30\" },\n    @{ Name = \"DailyDigest\";   Script = \"ops\\digest.py\";      Time = \"07:00\" }\n)\n\nforeach ($t in $tasks) {\n    $scriptPath = Join-Path $Root $t.Script\n    $action = New-ScheduledTaskAction `\n        -Execute $python `\n        -Argument \"`\"$scriptPath`\"\" `\n        -WorkingDirectory (Split-Path $scriptPath)\n\n    $trigger = New-ScheduledTaskTrigger -Daily -At $t.Time\n\n    $settings = New-ScheduledTaskSettingsSet `\n        -AllowStartIfOnBatteries `\n        -DontStopIfGoingOnBatteries `\n        -StartWhenAvailable `\n        -ExecutionTimeLimit (New-TimeSpan -Hours 2)\n\n    Register-ScheduledTask `\n        -TaskName $t.Name `\n        -Action $action `\n        -Trigger $trigger `\n        -Settings $settings `\n        -User $env:USERNAME `\n        -RunLevel Highest `\n        -Force\n}\n\nWrite-Host \"Registered $($tasks.Count) tasks\"\n```\n\n`-StartWhenAvailable` matters on a server that may be asleep or rebooting at the trigger time. `-ExecutionTimeLimit` prevents a hung script from blocking the next day's run. `-Force` makes the script idempotent, so re-running it after a code update just overwrites the existing task.\n\n## The encoding trap\n\nThis is where the migration nearly failed. The dev box ran Windows 11 with UTF-8 as the default code page. The server ran Windows Server 2016 with GBK (code page 936). The same Python scripts produced different output on each machine.\n\nThe symptoms were subtle. A blog post with a curly apostrophe rendered as `’` on the server. A trading log line with a Chinese instrument name raised `UnicodeEncodeError` when redirected to a file. A JSON state file written on the dev box loaded fine on the server, but a state file written on the server and read back on the dev box returned mojibake.\n\nThe root cause: Python 3.7+ uses `locale.getpreferredencoding()` for `open()` when no encoding is specified, and `sys.stdout` on Windows uses the console code page unless redirected. On the dev box that was UTF-8. On the server it was GBK. Every unqualified `open()` and every `print()` was a latent bug.\n\nThe fix is to pin encoding everywhere. No exceptions.\n\n### Pinning file I/O\n\n```python\nimport json\nfrom pathlib import Path\n\ndef read_json(path: Path) -\u003e dict:\n    with path.open(\"r\", encoding=\"utf-8\") as f:\n        return json.load(f)\n\ndef write_json(path: Path, data: dict) -\u003e None:\n    tmp = path.with_suffix(path.suffix + \".tmp\")\n    with tmp.open(\"w\", encoding=\"utf-8\", newline=\"\\n\") as f:\n        json.dump(data, f, ensure_ascii=False, indent=2)\n    tmp.replace(path)\n```\n\nThe `newline=\"\\n\"` matters when the same files are read by tools on other platforms. `ensure_ascii=False` keeps non-ASCII characters readable in the file instead of escaping them, which is fine as long as the encoding is pinned on both ends.\n\n### Pinning stdout\n\nRedirected stdout on Windows inherits the console code page, which is why logging to a file behaved differently from logging to the terminal. Reconfigure the streams at process start:\n\n```python\nimport sys\n\ndef force_utf8_streams() -\u003e None:\n    for stream in (sys.stdout, sys.stderr):\n        if hasattr(stream, \"reconfigure\"):\n            stream.reconfigure(encoding=\"utf-8\", errors=\"replace\")\n\nforce_utf8_streams()\n```\n\nCall this before any imports that might log at module level. `errors=\"replace\"` prevents a single bad byte from killing a long-running task; log the replacement character and move on.\n\n### Pinning subprocess output\n\nAny `subprocess.run` that captures text inherits the same problem. Pass the encoding explicitly:\n\n```python\nimport subprocess\n\nresult = subprocess.run(\n    [\"git\", \"log\", \"-1\", \"--format=%s\"],\n    capture_output=True,\n    text=True,\n    encoding=\"utf-8\",\n    errors=\"replace\",\n    check=False,\n)\n```\n\nWithout `encoding=\"utf-8\"`, Python decodes the child process output using the locale code page. On the server that meant GBK decoding of UTF-8 bytes, which silently produced garbage in commit messages.\n\n### Pinning the environment\n\nFor belt and braces, set `PYTHONUTF8=1` in the task environment. Python 3.7+ honors this and enables UTF-8 mode globally, which makes `open()` default to UTF-8 and `sys.stdout` use UTF-8 regardless of the console code page:\n\n```powershell\n$env:PYTHONUTF8 = \"1\"\n```\n\nI set this in the scheduled task definition via the `-Environment` parameter on `New-ScheduledTaskAction` where supported, and as a system environment variable on the server otherwise. It does not replace explicit encoding arguments in code, but it catches anything I missed.\n\n## The verification bat\n\nAfter install, `verify.bat` checks every task's action and last run result. This caught two tasks that had been registered with the wrong Python path and one that had never run because its trigger time had already passed on the install day:\n\n```bat\n@echo off\nsetlocal enabledelayedexpansion\n\nset \"FAIL=0\"\n\nfor %%T in (\n    BlogPublish BlogFetch TradeScan TradeExecute\n    TradeReport StateBackup HealthCheck LogRotate DailyDigest\n) do (\n    for /f \"delims=\" %%A in ('powershell -NoProfile -Command ^\n        \"(Get-ScheduledTask -TaskName '%%T').Actions[0].Execute\"') do (\n        set \"EXEC=%%A\"\n    )\n    for /f \"delims=\" %%R in ('powershell -NoProfile -Command ^\n        \"(Get-ScheduledTaskInfo -TaskName '%%T').LastTaskResult\"') do (\n        set \"RESULT=%%R\"\n    )\n\n    echo %%T  exec=!EXEC!  last=!RESULT!\n\n    if \"!EXEC!\"==\"\" (\n        echo   MISSING ACTION\n        set \"FAIL=1\"\n    )\n    if not \"!RESULT!\"==\"0\" if not \"!RESULT!\"==\"267009\" (\n        echo   NONZERO RESULT\n        set \"FAIL=1\"\n    )\n)\n\nif \"!FAIL!\"==\"1\" (\n    echo Verification failed\n    exit /b 1\n)\necho All tasks verified\nendlocal\n```\n\n`267009` is the \"task is currently running\" result code, which is not a failure. Any other nonzero `LastTaskResult` gets flagged. The action check catches the case where a task exists but points at a Python interpreter that was uninstalled or moved.\n\n## What I would do differently\n\nTwo things. First, set `PYTHONUTF8=1` on both machines before writing any code, not after the first mojibake appears. Second, treat the exclude list as part of the source tree and review it whenever a new runtime directory is added; a stale exclude list is how you accidentally ship a trade record to a new machine.\n\nThe migration itself ran in about ten minutes once the package was built. Building the package took a day, most of it spent on the encoding audit. The scripts were never the problem. The environment was.\n\nMore notes like this ship every week on this site.\n\n---\n\n## Daily Picks\n\n\u003e The following pairs are selected from the multi-timeframe trend scanner (Gate.io futures) and are for technical-analysis study only — not investment advice.\n\u003e Data updated: 2026-10-06 12:36:33\n\n### Long\n\n| Pair | Signal | Price | Take Profit | Stop Loss | R/R |\n|------|--------|-------|-------------|-----------|-----|\n| SKYAI |  | $0.0416 | $0.0433 | $0.0406 | 1:1.6 |\n| RE |  | $0.4991 | $0.5181 | $0.4866 | 1:1.5 |\n\n_2 picks selected. Scanner runs every 15 minutes._\n","positive_reactions_count":0,"cover_image":null,"tag_list":["migration","xcopy","checklist","windowsserver"],"canonical_url":"https://www.hkcode.dpdns.org/en/posts/6.html","reading_time_minutes":7,"user":{"name":"mattleeee","username":"mattleeee","twitter_username":null,"github_username":"mattleeee","user_id":4160029,"website_url":null,"profile_image":"https://media2.dev.to/dynamic/image/width=640,height=640,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png","profile_image_90":"https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png"}},{"type_of":"article","id":4804468,"title":"Exit Code 267009: Debugging CMD Encoding Corruption","description":"Scheduled tasks fail in the dark. There is no console, no visible stderr, and often no useful log —...","published":true,"published_at":"2026-10-06T04:38:27.927Z","slug":"exit-code-267009-debugging-cmd-encoding-corruption-1eph","path":"/mattleeee/exit-code-267009-debugging-cmd-encoding-corruption-1eph","url":"https://dev.to/mattleeee/exit-code-267009-debugging-cmd-encoding-corruption-1eph","comments_count":0,"public_reactions_count":0,"page_views_count":60,"published_timestamp":"2026-10-06T04:38:27Z","body_markdown":"Scheduled tasks fail in the dark. There is no console, no visible stderr, and often no useful log — just a status code in Task Scheduler's \"Last Run Result\" column. When that code is `267009`, the usual suspects (missing file, wrong working directory, permissions) all look wrong, because the task itself is fine. What is broken is the *encoding* of the launcher script.\n\nThis post walks through a real failure: a `.bat` file saved as UTF-8 containing Chinese comments and paths, executed by `cmd.exe` under a GBK system codepage. We'll reproduce it, decode the exit code, and lay out a checklist for batch files that need to survive being copied between machines.\n\n## The Symptom\n\nA scheduled task ran a small batch launcher:\n\n```bat\n@echo off\nREM 启动数据抓取流程\ncd /d \"C:\\项目\\抓取\"\npython fetch.py --output \"C:\\项目\\数据\\out.csv\"\n```\n\nThe task's \"Last Run Result\" showed `267009 (0x41301)`. The Python script ran fine when invoked manually from a terminal. Under the scheduler, it never produced output and never wrote a log line. Nothing in Event Viewer beyond the exit code.\n\nThat `0x41301` is the give-away. So the first move is not to touch the script — it's to learn what the code actually means.\n\n## Reading Opaque CMD Exit Codes\n\nWindows exit codes are 32-bit values. Task Scheduler displays them in decimal and hex. When a code looks arbitrary, check the hex form against the documented `HRESULT`/`Win32` ranges:\n\n- `0x00000000`–`0x0000FFFF`: normal Win32 error codes (`ERROR_*`).\n- `0x80000000`–`0xFFFFFFFF`: `HRESULT`-style values, usually from COM or the shell.\n- `0x41301` specifically: `SCHED_S_TASK_RUNNING` — the task is still running.\n\nThat last one is the trap. `267009` is not an error at all. It is `SCHED_S_TASK_RUNNING`, a success-severity status meaning the task was *still executing* when the query was made. Task Scheduler reports it when:\n\n1. The task genuinely takes a long time and the status snapshot was taken mid-run.\n2. The launched process never exits — it hangs, usually waiting on input or blocked on a parse error.\n\nIn this case it was case 2: `cmd.exe` hit a malformed line, dropped into an interactive prompt, and sat there forever. The scheduled task \"ran,\" produced no output, and never completed. `267009` was the only clue.\n\nA quick way to normalize and classify codes before you go down a rabbit hole:\n\n```python\nimport ctypes\nfrom ctypes import wintypes\n\ndef describe_exit_code(code: int) -\u003e str:\n    \"\"\"Classify a Windows exit code into its documented family.\"\"\"\n    code \u0026= 0xFFFFFFFF  # normalize to unsigned 32-bit\n\n    if code == 0:\n        return \"success\"\n    if code == 0x41301:\n        return \"SCHED_S_TASK_RUNNING (task still executing or hung)\"\n    if code \u003c 0x10000:\n        msg = ctypes.FormatError(code).strip()\n        return f\"Win32 error {code} ({msg})\" if msg else f\"Win32 error {code}\"\n    if code \u003e= 0x80000000:\n        return f\"HRESULT 0x{code:08X} (facility {code \u003e\u003e 16 \u0026 0x1FFF})\"\n    return f\"unknown 0x{code:08X}\"\n\nfor c in (0, 267009, 2, 0x80070002, 1):\n    print(f\"{c:\u003e10} 0x{c \u0026 0xFFFFFFFF:08X}  {describe_exit_code(c)}\")\n```\n\nOutput:\n\n```plaintext\n         0 0x00000000  success\n    267009 0x00041301  SCHED_S_TASK_RUNNING (task still executing or hung)\n         2 0x00000002  Win32 error 2 (The system cannot find the file specified.)\n  2147942402 0x80070002  HRESULT 0x80070002 (facility 7)\n         1 0x00000001  Win32 error 1 (Incorrect function.)\n```\n\nTwo rules of thumb fall out of this:\n\n- If you see `0x41301`, the task is not failing — it is *not finishing*. Go look for a hang, not a crash.\n- `0x8007xxxx` is a Win32 error wrapped in an `HRESULT`; strip the `0x8007` prefix and you get the plain `ERROR_*` code. `0x80070002` is just \"file not found.\"\n\n## Reproducing the Corruption\n\nThe hang was caused by encoding, not logic. Here is the minimal reproduction.\n\nTake a `.bat` file with non-ASCII content — Chinese comments, a Chinese path, or both — and save it as **UTF-8 without BOM**. On a machine whose active code page is GBK (`chcp` returns `936`), run it:\n\n```bat\n@echo off\nREM 抓取任务\necho starting\ncd /d \"C:\\项目\\数据\"\necho done\n```\n\nUnder GBK, `cmd.exe` reads each byte of the UTF-8 sequence as a separate GBK character. UTF-8 encodes `项` as `E9 A1 B9`. GBK sees `E9 A1` as one character and `B9` as the start of another, consuming the next byte — which may be a quote, a newline, or part of `echo`. The parse decouples from the intended text.\n\nThe failure modes are consistent and annoying:\n\n- A `REM` comment line absorbs the following line, silently dropping a command.\n- A quoted path loses its closing quote, so `cmd.exe` keeps reading.\n- The script reaches end-of-file mid-command and `cmd.exe` falls back to reading from stdin — which, under a scheduled task, is an empty pipe. It blocks. Forever. Hence `267009`.\n\nYou can reproduce this deterministically without the scheduler:\n\n```python\nimport subprocess\n\n# A launcher with a UTF-8 Chinese comment, no BOM.\nbat_utf8 = (\n    \"@echo off\\r\\n\"\n    \"REM 抓取任务\\r\\n\"\n    \"echo starting\\r\\n\"\n    \"echo done\\r\\n\"\n)\n\nwith open(\"probe_utf8.bat\", \"wb\") as f:\n    f.write(bat_utf8.encode(\"utf-8\"))\n\n# Run it under a GBK console code page, capturing stdin from an empty pipe\n# so a hang is observable as a timeout rather than a block.\ntry:\n    proc = subprocess.run(\n        [\"cmd.exe\", \"/c\", \"chcp 936 \u003enul \u0026 probe_utf8.bat\"],\n        stdin=subprocess.DEVNULL,\n        capture_output=True,\n        timeout=5,\n    )\n    print(\"returncode:\", proc.returncode)\n    print(\"stdout:\", proc.stdout.decode(\"gbk\", \"replace\"))\nexcept subprocess.TimeoutExpired:\n    print(\"HANG: cmd.exe is waiting on stdin (this is the 267009 case)\")\n```\n\nOn a GBK system this prints `HANG`. The same content saved as GBK runs cleanly. That difference is the whole bug.\n\n## Why UTF-8 \"Works\" Until It Doesn't\n\nModern Windows 10/11 still defaults `cmd.exe` to the OEM code page — `936` (GBK) on Simplified Chinese installs, `437`/`850` on US installs — unless you enable the \"Beta: Use Unicode UTF-8 for worldwide language support\" option. Editors, by contrast, default to UTF-8. So the file is written in one encoding and read in another. It works on the developer's machine only if that machine happens to be configured the same way, which is exactly the kind of coincidence that breaks when you copy the file to a server or a VM.\n\nThe scheduled task made it worse: the interactive terminal had a code page that happened to tolerate the file, while the scheduler ran `cmd.exe` with no console and the system default. Same file, different reader.\n\n## The Fixes\n\nThree changes, applied together, kill the class of bug. None of them require the \"UTF-8 beta\" checkbox.\n\n### 1. Keep `.bat` files pure ASCII\n\nThe most robust launcher contains only ASCII bytes: no comments in Chinese, no non-ASCII paths inline. If you need a localized comment, put it in a `.txt` sidecar or in the Python script, not in the batch file. `cmd.exe` has no encoding problem with ASCII because ASCII is a subset of nearly every code page in use.\n\n### 2. If non-ASCII is unavoidable, save as GBK and use CRLF\n\nIf a path genuinely must appear literally, encode the file in the system's OEM code page (GBK on Chinese Windows) rather than UTF-8, and terminate lines with `\\r\\n`. `cmd.exe`'s parser treats a bare `\\n` inconsistently across versions; CRLF is what it expects. Do not add a UTF-8 BOM — `cmd.exe` will try to execute the BOM bytes as part of the first command.\n\n```python\ndef write_bat(path: str, text: str, encoding: str = \"gbk\") -\u003e None:\n    \"\"\"Write a .bat file cmd.exe can parse.\n\n    - encoding: OEM code page of the target machine (gbk, cp437, ...)\n    - line endings forced to CRLF\n    - no BOM\n    \"\"\"\n    normalized = text.replace(\"\\r\\n\", \"\\n\").replace(\"\\r\", \"\\n\")\n    data = normalized.replace(\"\\n\", \"\\r\\n\").encode(encoding, errors=\"strict\")\n    with open(path, \"wb\") as f:\n        f.write(data)  # never a BOM\n```\n\n`errors=\"strict\"` is deliberate: if a character cannot be represented in GBK, you want the write to fail loudly rather than silently replace it with `?` and produce a path that doesn't exist.\n\n### 3. Use 8.3 short paths for non-ASCII directories\n\nEven with correct encoding, non-ASCII paths are fragile across tools, quoting layers, and remote sessions. Windows exposes an ASCII-only 8.3 alias for every path. Resolve it once and put the short form in the launcher:\n\n```python\nimport ctypes\nfrom ctypes import wintypes\n\ndef short_path(long_path: str) -\u003e str:\n    \"\"\"Return the 8.3 short form of a path (ASCII-only).\"\"\"\n    GetShortPathNameW = ctypes.windll.kernel32.GetShortPathNameW\n    GetShortPathNameW.argtypes = [wintypes.LPCWSTR, wintypes.LPWSTR, wintypes.DWORD]\n    GetShortPathNameW.restype = wintypes.DWORD\n\n    buf = ctypes.create_unicode_buffer(260)\n    n = GetShortPathNameW(long_path, buf, len(buf))\n    if n == 0:\n        raise OSError(ctypes.get_last_error(), f\"no short path for {long_path!r}\")\n    if n \u003e len(buf):\n        buf = ctypes.create_unicode_buffer(n)\n        GetShortPathNameW(long_path, buf, n)\n    return buf.value\n\nprint(short_path(r\"C:\\项目\\数据\"))  # e.g. C:\\A1B2~1\\A3C4~1\n```\n\nThe generated launcher is now pure ASCII:\n\n```bat\n@echo off\ncd /d \"C:\\A1B2~1\\A3C4~1\"\npython fetch.py --output \"C:\\A1B2~1\\A3C4~1\\out.csv\"\n```\n\n8.3 name generation can be disabled per-volume (`fsutil 8dot3name query`), so verify it is enabled on volumes where you rely on this. If it is off, fall back to fix 1 or 2.\n\n## A Checklist for Portable `.bat` Files\n\nBefore a batch file goes into a scheduled task or gets copied to another machine, run through this:\n\n1. **Bytes are ASCII-only.** Verify with a script, not by eye — invisible non-ASCII is easy to miss.\n2. **Line endings are CRLF.** No lone `\\n`, no lone `\\r`.\n3. **No BOM.** The first three bytes must not be `EF BB BF`.\n4. **No non-ASCII paths inline.** Use 8.3 short paths or pass paths as arguments from a caller that handles Unicode.\n5. **Encoding matches the target.** If ASCII is impossible, the file is GBK (or the target's OEM code page), not UTF-8.\n6. **`chcp` is explicit if needed.** If you must change the code page, do it as the first line and document why.\n7. **No interactive fallback.** End with an explicit `exit /b %ERRORLEVEL%` so `cmd.exe` never drops into a prompt on parse failure.\n8. **Test under the scheduler's environment.** A terminal session and a scheduled task are not the same reader.\n\nA guard you can run in CI or as a pre-commit hook:\n\n```python\ndef audit_bat(path: str) -\u003e list[str]:\n    problems = []\n    raw = open(path, \"rb\").read()\n\n    if raw.startswith(b\"\\xef\\xbb\\xbf\"):\n        problems.append(\"has UTF-8 BOM\")\n    if b\"\\r\\n\" not in raw:\n        problems.append(\"no CRLF line endings\")\n    if raw.replace(b\"\\r\\n\", b\"\").count(b\"\\n\") or raw.replace(b\"\\r\\n\", b\"\").count(b\"\\r\"):\n        problems.append(\"mixed or bare line endings\")\n    try:\n        raw.decode(\"ascii\")\n    except UnicodeDecodeError as e:\n        problems.append(f\"non-ASCII byte at offset {e.start}: {raw[e.start:e.start+1]!r}\")\n    return problems\n\nfor p in (\"probe_utf8.bat\", \"launcher.bat\"):\n    issues = audit_bat(p)\n    print(p, \"OK\" if not issues else issues)\n```\n\nRun it against the original failing launcher and it reports the non-ASCII byte immediately — the same byte `cmd.exe` choked on.\n\n## Takeaway\n\n`267009` is not an error code in the usual sense; it is `SCHED_S_TASK_RUNNING`, and it most often means `cmd.exe` is blocked on a parse failure that turned into a stdin read. The parse failure came from an encoding mismatch between the file (UTF-8) and the reader (GBK). Fix the file's bytes — ASCII where possible, GBK where not, CRLF always, no BOM — and use 8.3 short paths for anything non-ASCII. The exit code disappears because there is nothing left for `cmd.exe` to misread.\n\nMore notes like this ship every week on this site.\n\n---\n\n## Daily Picks\n\n\u003e The following pairs are selected from the multi-timeframe trend scanner (Gate.io futures) and are for technical-analysis study only — not investment advice.\n\u003e Data updated: 2026-10-06 12:36:33\n\n### Long\n\n| Pair | Signal | Price | Take Profit | Stop Loss | R/R |\n|------|--------|-------|-------------|-----------|-----|\n| SKYAI |  | $0.0416 | $0.0433 | $0.0406 | 1:1.6 |\n| RE |  | $0.4991 | $0.5181 | $0.4866 | 1:1.5 |\n\n_2 picks selected. Scanner runs every 15 minutes._\n","positive_reactions_count":0,"cover_image":null,"tag_list":["exitcode","267009","cmdexe","encoding"],"canonical_url":"https://www.hkcode.dpdns.org/en/posts/5.html","reading_time_minutes":8,"user":{"name":"mattleeee","username":"mattleeee","twitter_username":null,"github_username":"mattleeee","user_id":4160029,"website_url":null,"profile_image":"https://media2.dev.to/dynamic/image/width=640,height=640,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png","profile_image_90":"https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png"}},{"type_of":"article","id":4804466,"title":"Staggering Nine Scheduled Tasks to Eliminate Resource Contention","description":"A market-signal job ran at HH:00. A data-refresh job ran at HH:00. Both touched the same JSON files...","published":true,"published_at":"2026-10-06T04:37:50.787Z","slug":"staggering-nine-scheduled-tasks-to-eliminate-resource-contention-522f","path":"/mattleeee/staggering-nine-scheduled-tasks-to-eliminate-resource-contention-522f","url":"https://dev.to/mattleeee/staggering-nine-scheduled-tasks-to-eliminate-resource-contention-522f","comments_count":0,"public_reactions_count":0,"page_views_count":40,"published_timestamp":"2026-10-06T04:37:50Z","body_markdown":"A market-signal job ran at `HH:00`. A data-refresh job ran at `HH:00`. Both touched the same JSON files and both wanted the CPU at the same instant. The result was predictable: `PermissionError` on Windows file locks, occasional half-written JSON that the signal job then parsed as empty, and a scheduler log full of overlapping runs.\n\nThe fix wasn't a faster machine or a rewrite. It was a timetable. This post walks through how I inventoried the triggers, decided on offsets, handled the shared-state file locks, and ended up with a single page that shows the whole fleet.\n\n## Why top-of-hour schedules collide\n\nCron-style schedulers are seductive because `0 * * * *` reads like \"do the thing.\" It doesn't say \"do the thing when nothing else is doing the thing.\" Every job author independently picks round numbers, and round numbers cluster. If nine jobs all live on `0 * * * *` or `0 4 * * *`, they share a trigger boundary whether they need to or not.\n\nTwo failure modes show up immediately:\n\n1. **CPU contention.** Two CPU-bound jobs start together, each takes roughly twice as long as it would alone, and the overlap window grows.\n2. **File-lock collisions.** On Windows, opening a file for write while another process holds it raises `PermissionError` or silently truncates depending on the flags. Two writers to the same JSON file is a data-corruption bug waiting for a slow disk.\n\nThe signal job and the refresh job hit both. The signal job read `signals.json` and wrote `positions.json`. The refresh job wrote `signals.json`. When they ran together, the signal job sometimes read an empty file mid-write and produced an empty position list — which then propagated downstream.\n\n## Step 1: Inventory every trigger\n\nYou can't stagger what you haven't listed. I dumped every scheduled entry into one place. On Windows, the Task Scheduler is scriptable through `schtasks` or the COM interface. The COM path gives structured data without parsing text.\n\n```python\nimport win32com.client\nfrom datetime import datetime\n\ndef list_scheduled_tasks(folder_path=\"\\\\\"):\n    \"\"\"Walk a Task Scheduler folder and return trigger summaries.\"\"\"\n    scheduler = win32com.client.Dispatch(\"Schedule.Service\")\n    scheduler.Connect()\n    root = scheduler.GetFolder(folder_path)\n    tasks = root.GetTasks(1)  # 1 = include hidden\n\n    rows = []\n    for task in tasks:\n        definition = task.Definition\n        for trigger in definition.Triggers:\n            rows.append({\n                \"name\": task.Name,\n                \"enabled\": task.Enabled,\n                \"type\": trigger.Type,\n                \"start\": getattr(trigger, \"StartBoundary\", None),\n                \"repetition\": getattr(trigger, \"Repetition\", None),\n            })\n    return rows\n\nfor row in list_scheduled_tasks():\n    print(f\"{row['name']:\u003c40} {row['start']}\")\n```\n\nThat gives you names and start boundaries. It doesn't tell you what the jobs *do* — which is the part that matters for spacing. So I added a second pass: for each task, record the action (the script path) and a manual tag for CPU-bound vs I/O-bound.\n\n```python\nTAGS = {\n    \"MarketSignalHourly\":   {\"kind\": \"cpu\", \"reads\": [\"signals.json\"],\n                             \"writes\": [\"positions.json\"]},\n    \"DataRefreshHourly\":    {\"kind\": \"io\",  \"reads\": [],\n                             \"writes\": [\"signals.json\"]},\n    \"NightlyMaintenance\":   {\"kind\": \"cpu\", \"reads\": [\"positions.json\"],\n                             \"writes\": [\"positions.json\", \"audit.log\"]},\n    \"BlogPublish\":          {\"kind\": \"io\",  \"reads\": [\"posts/*.md\"],\n                             \"writes\": [\"site/\"]},\n    \"BlogPublishDE\":        {\"kind\": \"io\",  \"reads\": [\"posts/de/*.md\"],\n                             \"writes\": [\"site/de/\"]},\n}\n```\n\nThe `reads`/`writes` sets are the important column. Two jobs that share a write target must never overlap, regardless of how cheap they are.\n\n## Step 2: Rules of thumb for spacing\n\nOnce you have the inventory, spacing is mostly mechanical. Three rules cover almost everything:\n\n### CPU-bound jobs need a full estimated runtime of separation\n\nIf a job takes 12 minutes of CPU on a busy box, don't schedule the next CPU-bound job 5 minutes later. Measure the p95 runtime, not the median. The overlap you're trying to avoid is the tail, not the typical case.\n\n### I/O-bound jobs can share a slot if they touch different files\n\nTwo jobs that read disjoint file sets and write to disjoint outputs can run together. Disk throughput is usually fine for a handful of readers. The moment they write the same path, they can't share.\n\n### Never let a writer and a reader of the same file overlap\n\nThis is the hard rule. Even a fast reader can catch a partial write. Either serialize them or make the write atomic (see the next section).\n\nApplying these to the fleet:\n\n| Job | Kind | Runtime (p95) | Constraint |\n|---|---|---|---|\n| MarketSignalHourly | cpu | 14 min | writes `positions.json` |\n| DataRefreshHourly | io | 6 min | writes `signals.json` |\n| NightlyMaintenance | cpu | 40 min | writes `positions.json`, `audit.log` |\n| BlogPublish | io | 3 min | writes `site/` |\n| BlogPublishDE | io | 3 min | writes `site/de/` |\n\nThe signal job and the refresh job both ran at `HH:00`. The signal job reads `signals.json`; the refresh job writes it. That's the writer/reader overlap, and it's the one that corrupted data. The fix: move the refresh job off the top of the hour so it finishes before the signal job starts, or at least so the write window doesn't intersect the read window.\n\nI moved the refresh to `HH:45` — 15 minutes before the signal job. The refresh takes 6 minutes at p95, so it's done by `HH:51`, leaving a 9-minute buffer. That buffer absorbs the tail.\n\n## Step 3: The offset schedule\n\nHere's the timetable that emerged. The signal job stays at `HH:00` because downstream consumers expect it there. Everything else moves around it.\n\n```plaintext\nHH:00  MarketSignalHourly      (cpu, 14 min)\nHH:45  DataRefreshHourly       (io, 6 min)  -\u003e completes before next HH:00\n04:30  NightlyMaintenance      (cpu, 40 min) -\u003e done by 05:10\n09:00  BlogPublish             (io, 3 min)\n10:30  BlogPublishDE           (io, 3 min)\n```\n\nThe daily jobs sit in a quiet window. `04:30` is after the overnight batch and before the morning signal jobs ramp up. `09:00` and `10:30` are separated by 90 minutes, which is overkill for two 3-minute jobs, but the gap gives room for retries and for the English site build to finish before the German one starts — they share a `site/` parent directory, so a partial build on one side can confuse a deploy that scans the whole tree.\n\nThe `HH:45` offset is the one that matters most. It's not a round number, which is the point. Round numbers cluster; odd offsets don't.\n\n## Step 4: Handle the shared state file\n\nStaggering reduces overlap but doesn't eliminate it. A job that overruns its window will still collide with the next one. The durable fix is atomic writes plus a lock file.\n\nOn Windows, `os.replace` is atomic on the same volume. Write to a temp file, then replace the target. Readers either see the old file or the new one, never a partial write.\n\n```python\nimport json\nimport os\nimport tempfile\nfrom pathlib import Path\n\ndef atomic_write_json(path: Path, payload: dict) -\u003e None:\n    \"\"\"Write JSON so readers never observe a partial file.\"\"\"\n    path = Path(path)\n    fd, tmp_name = tempfile.mkstemp(\n        dir=path.parent, prefix=path.name + \".\", suffix=\".tmp\"\n    )\n    try:\n        with os.fdopen(fd, \"w\", encoding=\"utf-8\") as fh:\n            json.dump(payload, fh)\n            fh.flush()\n            os.fsync(fh.fileno())\n        os.replace(tmp_name, path)  # atomic on same volume\n    except BaseException:\n        os.unlink(tmp_name)\n        raise\n```\n\nThat handles readers. For writers, add a lock so two processes can't both decide to write. A simple lock file with an exclusive create works on Windows and POSIX:\n\n```python\nimport time\nfrom contextlib import contextmanager\n\n@contextmanager\ndef file_lock(path: Path, timeout: float = 30.0):\n    \"\"\"Exclusive lock via atomic create. Windows-safe.\"\"\"\n    lock = Path(str(path) + \".lock\")\n    deadline = time.monotonic() + timeout\n    while True:\n        try:\n            fd = os.open(lock, os.O_CREAT | os.O_EXCL | os.O_WRONLY)\n            os.close(fd)\n            break\n        except FileExistsError:\n            if time.monotonic() \u003e deadline:\n                raise TimeoutError(f\"lock held: {lock}\")\n            time.sleep(0.5)\n    try:\n        yield\n    finally:\n        lock.unlink(missing_ok=True)\n```\n\nThen any job that writes `signals.json` wraps its write:\n\n```python\nwith file_lock(SIGNALS_PATH):\n    data = load_json(SIGNALS_PATH)\n    data.update(new_signals)\n    atomic_write_json(SIGNALS_PATH, data)\n```\n\nThe lock serializes writers; the atomic replace protects readers. Together they make the schedule forgiving of small overruns. Staggering is still the first line of defense — locks are the second.\n\nOne caveat: a stale lock file from a crashed process will block everything. Either add a PID check or a timestamp inside the lock file and expire it. For a small fleet, a timestamp is enough:\n\n```python\ndef lock_is_stale(lock: Path, max_age: float = 300.0) -\u003e bool:\n    try:\n        age = time.time() - lock.stat().st_mtime\n    except FileNotFoundError:\n        return False\n    return age \u003e max_age\n```\n\nCheck `lock_is_stale` before raising `TimeoutError`, and unlink if it's old.\n\n## Step 5: One page for the whole fleet\n\nThe inventory script and the tags table are useful, but they drift. The thing that actually kept the schedule sane was a single generated page — a timetable that lists every job, its trigger, its kind, and its file dependencies. I regenerate it from the source of truth (the task definitions plus the tags dict) and commit it.\n\n```python\ndef render_timetable(tasks, tags) -\u003e str:\n    lines = [\"| Job | Trigger | Kind | Reads | Writes |\", \"|---|---|---|---|---|\"]\n    for task in sorted(tasks, key=lambda t: t[\"start\"]):\n        tag = tags.get(task[\"name\"], {})\n        lines.append(\n            f\"| {task['name']} | {task['start']} | {tag.get('kind', '?')} \"\n            f\"| {', '.join(tag.get('reads', [])) or '-'} \"\n            f\"| {', '.join(tag.get('writes', [])) or '-'} |\"\n        )\n    return \"\\n\".join(lines)\n```\n\nThe value isn't the table itself — it's that adding a job forces you to fill in the `reads`/`writes` columns. That's the moment you notice two jobs want the same file at the same time. The timetable makes the collision visible before it happens, instead of after a corrupted JSON file shows up in the logs.\n\n## What changed\n\nAfter the rebalance, the `PermissionError` entries stopped. The signal job's parse failures went to zero because `signals.json` is now written by a job that finishes 9 minutes before the signal job reads it, and even if it overruns, the atomic write means the signal job reads a complete file. The nightly maintenance window moved off the top of the hour, so it no longer competes with the first morning signal run.\n\nNone of this required new hardware. It required looking at the schedule as a system instead of a list of independent jobs, measuring the tail runtimes, and picking offsets that don't cluster on round numbers.\n\nMore notes like this ship every week on this site.\n\n---\n\n## Daily Picks\n\n\u003e The following pairs are selected from the multi-timeframe trend scanner (Gate.io futures) and are for technical-analysis study only — not investment advice.\n\u003e Data updated: 2026-10-06 12:36:33\n\n### Long\n\n| Pair | Signal | Price | Take Profit | Stop Loss | R/R |\n|------|--------|-------|-------------|-----------|-----|\n| SKYAI |  | $0.0416 | $0.0433 | $0.0406 | 1:1.6 |\n| RE |  | $0.4991 | $0.5181 | $0.4866 | 1:1.5 |\n\n_2 picks selected. Scanner runs every 15 minutes._\n","positive_reactions_count":0,"cover_image":null,"tag_list":["staggering","taskscheduling","resourcecontention","cronlikedesign"],"canonical_url":"https://www.hkcode.dpdns.org/en/posts/4.html","reading_time_minutes":7,"user":{"name":"mattleeee","username":"mattleeee","twitter_username":null,"github_username":"mattleeee","user_id":4160029,"website_url":null,"profile_image":"https://media2.dev.to/dynamic/image/width=640,height=640,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png","profile_image_90":"https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png"}},{"type_of":"article","id":4804430,"title":"When GitHub Is Blocked but You Still Need to Push","description":"Corporate firewalls are rarely all-or-nothing. They are usually the result of a proxy policy, a TLS...","published":true,"published_at":"2026-10-06T04:29:02.647Z","slug":"when-github-is-blocked-but-you-still-need-to-push-43g7","path":"/mattleeee/when-github-is-blocked-but-you-still-need-to-push-43g7","url":"https://dev.to/mattleeee/when-github-is-blocked-but-you-still-need-to-push-43g7","comments_count":0,"public_reactions_count":1,"page_views_count":42,"published_timestamp":"2026-10-06T04:29:02Z","body_markdown":"Corporate firewalls are rarely all-or-nothing. They are usually the result of a proxy policy, a TLS inspection rule, or a category filter that someone applied years ago and forgot about. That is why you can end up in a situation where `github.com:443` is silently dropped, while `api.github.com` and `raw.githubusercontent.com` answer requests without complaint.\n\nI ran into this on a Windows build box inside a restricted network. `git push` hung until timeout. `git clone` over HTTPS failed. But `curl.exe https://api.github.com` returned a JSON payload in under a second. The deployment pipeline needed to publish a handful of generated files to a repository, and the network had effectively amputated the git transport while leaving the REST API intact.\n\nThis post covers how to diagnose that split, how to push files through the GitHub Contents API instead of git, and why the HTTP-only path is worth keeping around even on networks where git works fine.\n\n## Probing the firewall with curl.exe\n\nBefore writing any Python, establish exactly which endpoints survive. On Windows 10 and 11, `curl.exe` ships with the OS, so you do not need to install anything or fight with the PowerShell `curl` alias (which is actually `Invoke-WebRequest` and behaves differently).\n\nTest the three hosts that matter:\n\n```powershell\ncurl.exe -sS -o NUL -w \"%{http_code} %{time_total}s\\n\" https://github.com\ncurl.exe -sS -o NUL -w \"%{http_code} %{time_total}s\\n\" https://api.github.com\ncurl.exe -sS -o NUL -w \"%{http_code} %{time_total}s\\n\" https://raw.githubusercontent.com\n```\n\n`-o NUL` discards the body, `-w` prints the status code and elapsed time. On the blocked network, the first command sat there for roughly 20 seconds and exited with code 28 (operation timed out). The other two returned `200` in a few hundred milliseconds.\n\nA timeout is more informative than a `403`. A `403` means something answered — a proxy, a filter appliance — and you can sometimes negotiate with it. A timeout means the connection was dropped or blackholed, which usually points to a routing or SNI-based block that you cannot talk your way out of.\n\nConfirm the git transport specifically:\n\n```powershell\ngit ls-remote https://github.com/your-org/your-repo.git\n```\n\nIf that hangs while `curl.exe` to `api.github.com` succeeds, you have confirmed the split: git's smart HTTP protocol talks to `github.com`, the API talks to `api.github.com`, and only one of them is reachable.\n\nOne caveat worth checking early: `raw.githubusercontent.com` serves file *content* but does not accept writes. It is useful for reads and for verifying that a push landed, but every write goes through `api.github.com`.\n\n## Authenticating without a browser\n\nThe Contents API needs a token. A classic personal access token with `repo` scope works, or a fine-grained token with `Contents: Read and write` on the target repository. Put it in an environment variable rather than the script:\n\n```powershell\n$env:GITHUB_TOKEN = \"ghp_...\"\n```\n\nThen verify the token and check your rate limit budget in one call:\n\n```powershell\ncurl.exe -sS -H \"Authorization: Bearer $env:GITHUB_TOKEN\" `\n  -H \"Accept: application/vnd.github+json\" `\n  https://api.github.com/rate_limit\n```\n\nThe response includes `resources.core.limit`, normally `5000`, and `remaining`. That number is per hour, per authenticated token. Unauthenticated requests get 60 per hour from a shared IP pool, which is useless for any real deployment — always authenticate.\n\n## The Contents API, step by step\n\nThe Contents API models a repository file as a resource at:\n\n```plaintext\n/repos/{owner}/{repo}/contents/{path}\n```\n\nA write is a two-step dance, because the API is optimistic-concurrency aware. You cannot just PUT a file and hope; if the file already exists, you must supply the current blob SHA so the server knows you are updating the version you think you are updating.\n\n### Step 1: GET the current SHA\n\n```python\nimport base64\nimport os\nimport time\n\nimport requests\n\nAPI = \"https://api.github.com\"\nOWNER = \"your-org\"\nREPO = \"your-repo\"\nBRANCH = \"main\"\nTOKEN = os.environ[\"GITHUB_TOKEN\"]\n\nHEADERS = {\n    \"Authorization\": f\"Bearer {TOKEN}\",\n    \"Accept\": \"application/vnd.github+json\",\n    \"X-GitHub-Api-Version\": \"2022-11-28\",\n}\n\n\ndef get_file_sha(path: str) -\u003e str | None:\n    url = f\"{API}/repos/{OWNER}/{REPO}/contents/{path}\"\n    r = requests.get(url, headers=HEADERS, params={\"ref\": BRANCH}, timeout=30)\n    if r.status_code == 404:\n        return None\n    r.raise_for_status()\n    return r.json()[\"sha\"]\n```\n\n`404` means the file does not exist yet — that is a create, not an update. Anything else non-2xx should raise, because silently continuing on a `401` or `403` just produces confusing failures later.\n\n### Step 2: PUT the base64 content\n\n```python\ndef put_file(path: str, content: bytes, message: str, sha: str | None) -\u003e dict:\n    url = f\"{API}/repos/{OWNER}/{REPO}/contents/{path}\"\n    payload = {\n        \"message\": message,\n        \"content\": base64.b64encode(content).decode(\"ascii\"),\n        \"branch\": BRANCH,\n    }\n    if sha is not None:\n        payload[\"sha\"] = sha\n\n    r = requests.put(url, headers=HEADERS, json=payload, timeout=30)\n    r.raise_for_status()\n    return r.json()\n```\n\nThe `content` field must be base64 with no line breaks. Python's `base64.b64encode` produces exactly that. If you are reading a text file, encode it to UTF-8 first — do not pass a `str` to `b64encode`, it will raise.\n\n### Step 3: One commit per file, with a sleep\n\nThe Contents API commits one file per request. There is no multi-file atomic commit through this endpoint; for that you would need the Git Data API (blobs, trees, commits, refs), which is more calls, not fewer. For a deployment that publishes a handful of generated artifacts, one commit per file is fine and much simpler to reason about.\n\n```python\ndef push_files(files: dict[str, bytes], message_prefix: str) -\u003e None:\n    for path, content in files.items():\n        sha = get_file_sha(path)\n        verb = \"Update\" if sha else \"Add\"\n        result = put_file(path, content, f\"{verb} {path}\", sha)\n        commit_sha = result[\"commit\"][\"sha\"][:7]\n        print(f\"{verb.lower():\u003e6} {path} -\u003e {commit_sha}\")\n        time.sleep(0.4)\n```\n\nThe `0.4` second sleep is not superstition. GitHub's secondary rate limits are separate from the 5000/hour primary budget and are triggered by bursty write patterns. A tight loop of PUTs to the same repository is exactly the shape that trips them. Four tenths of a second keeps you under the radar while still pushing a few hundred files in a couple of minutes. If you are pushing thousands of files, raise it or batch through the Git Data API instead.\n\nA `409` response on the PUT means the SHA you supplied is stale — someone else committed between your GET and your PUT. The correct handling is to re-GET and retry once:\n\n```python\ndef put_file_with_retry(path, content, message, attempts=3):\n    for i in range(attempts):\n        sha = get_file_sha(path)\n        try:\n            return put_file(path, content, message, sha)\n        except requests.HTTPError as e:\n            if e.response.status_code == 409 and i \u003c attempts - 1:\n                time.sleep(1.0)\n                continue\n            raise\n```\n\nDo not loop forever on `409`. If two processes are fighting over the same file, retrying just amplifies the conflict.\n\n## Verifying the push without git\n\nAfter the PUTs, confirm the content actually landed. `raw.githubusercontent.com` is reachable on this network and serves the file at the branch tip:\n\n```python\ndef verify(path: str, expected: bytes) -\u003e bool:\n    url = f\"https://raw.githubusercontent.com/{OWNER}/{REPO}/{BRANCH}/{path}\"\n    r = requests.get(url, timeout=30)\n    r.raise_for_status()\n    return r.content == expected\n```\n\nThere is a caching layer in front of `raw.githubusercontent.com` with a short TTL — usually a minute or two. If verification fails immediately after a push, wait and retry before assuming the push failed. The commit itself is authoritative; check `GET /repos/{owner}/{repo}/commits?path={path}` if you need certainty faster than the CDN refreshes.\n\n## Why this path is worth keeping\n\nThe obvious use case is a network that blocks `github.com` but not the API. That is what prompted this work. But the HTTP-only push path has a second, less obvious value: it works when git itself is broken.\n\nI have hit all of these on build machines:\n\n- A `git` binary too old to negotiate the current TLS or protocol version, with no admin rights to upgrade it.\n- A corrupted `~/.gitconfig` or credential helper that makes every git invocation fail before it reaches the network.\n- A container image where the git install is stripped down and `git push` fails on missing helpers.\n- A CI runner with a read-only filesystem except for a scratch directory, where git's lock files cannot be written.\n\nIn all of those cases, a Python script using `requests` and a token pushes files just fine, because it depends on nothing but the standard library and an HTTP client. The API surface is stable and versioned via the `X-GitHub-Api-Version` header. There is no local state to corrupt.\n\nThe trade-off is real: one commit per file, no atomic multi-file changes, no branches created in a single call, and a hard 100 MB limit per file (the API rejects blobs over that size). For deployment artifacts — config files, generated reports, small data files, documentation — none of that matters. For a large repository with binary assets, stick with git and fix the network instead.\n\n## Operational notes\n\nKeep the token out of the script and out of logs. `requests` will happily include it in an exception traceback if you print the request; log only the URL and status code.\n\nSet an explicit `timeout` on every call. The default is no timeout, and on a flaky network that means a hung process rather than a clean failure.\n\nIf you are pushing to a protected branch, the token's identity must be allowed to bypass the protection or have the right to push. The API enforces branch protection the same way git does, and a `422` on the PUT often means a required status check or review is missing, not that your payload is malformed. Read the `message` field in the response body — it usually says which rule blocked you.\n\nFinally, keep the probe script. Networks change. The day `github.com` comes back, `git push` is still the better tool, and you will want to know which path is live without re-deriving it from scratch.\n\nMore notes like this ship every week on this site.\n\n---\n\n## Daily Picks\n\n\u003e The following pairs are selected from the multi-timeframe trend scanner (Gate.io futures) and are for technical-analysis study only — not investment advice.\n\u003e Data updated: 2026-10-06 12:36:33\n\n### Long\n\n| Pair | Signal | Price | Take Profit | Stop Loss | R/R |\n|------|--------|-------|-------------|-----------|-----|\n| SKYAI |  | $0.0416 | $0.0433 | $0.0406 | 1:1.6 |\n| RE |  | $0.4991 | $0.5181 | $0.4866 | 1:1.5 |\n\n_2 picks selected. Scanner runs every 15 minutes._\n","positive_reactions_count":1,"cover_image":null,"tag_list":["githubapi","curlexe","rawgithubusercontentcom","firewall"],"canonical_url":"https://www.hkcode.dpdns.org/en/posts/10.html","reading_time_minutes":7,"user":{"name":"mattleeee","username":"mattleeee","twitter_username":null,"github_username":"mattleeee","user_id":4160029,"website_url":null,"profile_image":"https://media2.dev.to/dynamic/image/width=640,height=640,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png","profile_image_90":"https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png"}},{"type_of":"article","id":4804425,"title":"Taming BLAS Threads in Scheduled Python Jobs","description":"Scheduled Python jobs on a shared box have a nasty habit of fighting each other. You write a script...","published":true,"published_at":"2026-10-06T04:27:49.964Z","slug":"taming-blas-threads-in-scheduled-python-jobs-nka","path":"/mattleeee/taming-blas-threads-in-scheduled-python-jobs-nka","url":"https://dev.to/mattleeee/taming-blas-threads-in-scheduled-python-jobs-nka","comments_count":0,"public_reactions_count":0,"page_views_count":50,"published_timestamp":"2026-10-06T04:27:49Z","body_markdown":"Scheduled Python jobs on a shared box have a nasty habit of fighting each other. You write a script that runs in four seconds on your laptop, deploy it to a Windows Server with eight cores, and now it takes forty — but only when the nightly batch is also running. The culprit is usually not your code. It is the BLAS thread pool underneath numpy.\n\n## What Actually Spawns Threads\n\nWhen you `import numpy`, the wheel you installed is linked against a BLAS implementation: OpenBLAS for the standard `numpy` wheels on PyPI, MKL for `numpy` built against Intel's math library (common with conda, or `pip install mkl` setups). Both ship multithreaded kernels for `dgemm`, `sgemm`, and friends. By default they read the machine's core count and spin up a pool of that size.\n\nSo `import numpy` on an 8-core server means eight OpenBLAS worker threads, plus the OpenMP runtime underneath, plus whatever your own `concurrent.futures` pool does. Matrix multiply is fast. Everything else pays for it.\n\nThe problem shows up when jobs overlap. Suppose Task Scheduler fires three jobs at 02:00:\n\n- Job A does a rolling regression over 200k rows.\n- Job B builds a covariance matrix.\n- Job C just reads a CSV with pandas and writes a parquet file.\n\nEach one imports numpy, each one gets eight BLAS threads. That is 24 threads on 8 cores. The OS context-switches between them, cache lines bounce between cores, and the memory allocator thrashes. Job C, which does almost no math, is now slow because it is competing for scheduler time with two BLAS pools it never asked for.\n\nThe fix is boring and effective: pin every job to one BLAS thread and let the OS schedule your *jobs* in parallel instead of your *math kernels*.\n\n## The Three Variables\n\n### OPENBLAS_NUM_THREADS\n\nRead by OpenBLAS at initialization. Controls the size of its internal thread pool. This is the one that matters for stock PyPI numpy wheels.\n\n### MKL_NUM_THREADS\n\nThe equivalent knob for Intel MKL. If your numpy is MKL-linked, this is the one that counts. MKL also honors `MKL_DYNAMIC` and `MKL_THREADING_LAYER`, but for our purposes `MKL_NUM_THREADS=1` is enough.\n\n### OMP_NUM_THREADS\n\nRead by the OpenMP runtime. Both OpenBLAS and MKL can use OpenMP as their threading backend, and so can other libraries (scikit-learn, some pandas paths, PyTorch). Setting this catches the cases where the BLAS-specific variable is ignored because the library was built with OpenMP threading.\n\nSet all three. The cost of setting one that is unused is zero; the cost of missing the one that matters is a mystery slowdown you will spend an afternoon debugging.\n\nThere is also `NUMEXPR_NUM_THREADS` for `numexpr`, which pandas uses for some `eval` paths, and `VECLIB_MAXIMUM_THREADS` on macOS. On Windows servers the first three cover the realistic cases.\n\n## Where to Set Them\n\nTwo places, and they are not equivalent.\n\n### In the launcher\n\nThe safest place is the process environment before Python starts. In a `.bat` file:\n\n```bat\n@echo off\nset OPENBLAS_NUM_THREADS=1\nset MKL_NUM_THREADS=1\nset OMP_NUM_THREADS=1\nset NUMEXPR_NUM_THREADS=1\n\"C:\\Python311\\python.exe\" \"D:\\jobs\\build_features.py\" \u003e\u003e \"D:\\logs\\build_features.log\" 2\u003e\u00261\n```\n\nThis is the version I use in Task Scheduler actions. The variables are set before `python.exe` is even loaded, so every library that reads them at import time gets the right value. No ordering concerns, no surprises.\n\nIf you drive jobs from PowerShell instead:\n\n```powershell\n$env:OPENBLAS_NUM_THREADS = \"1\"\n$env:MKL_NUM_THREADS      = \"1\"\n$env:OMP_NUM_THREADS      = \"1\"\n\u0026 \"C:\\Python311\\python.exe\" \"D:\\jobs\\build_features.py\"\n```\n\nSame effect. The environment block is inherited by the child process.\n\n### In Python, before importing numpy\n\nSometimes you cannot control the launcher — a CI runner, a wrapper someone else owns, a `python -m` invocation from a service. Then set the variables at the very top of your entry point:\n\n```python\n# entry.py — must run before any numerical import\nimport os\n\nos.environ.setdefault(\"OPENBLAS_NUM_THREADS\", \"1\")\nos.environ.setdefault(\"MKL_NUM_THREADS\", \"1\")\nos.environ.setdefault(\"OMP_NUM_THREADS\", \"1\")\nos.environ.setdefault(\"NUMEXPR_NUM_THREADS\", \"1\")\n\n# Now it is safe to import the numerical stack.\nimport numpy as np\nimport pandas as pd\n```\n\nTwo things to notice.\n\nFirst, `setdefault` rather than `=`. If the launcher already set a value, respect it. This lets you override per-job without editing the script.\n\nSecond, the ordering is load-bearing. OpenBLAS reads `OPENBLAS_NUM_THREADS` inside its constructor, which runs when the shared library is `dlopen`ed — i.e. during `import numpy`. If you set the variable after that import, you are too late. The pool is already sized.\n\nThis bites people who put the `os.environ` lines in a `config.py` that gets imported after numpy somewhere up the chain. If you cannot guarantee import order, use the launcher.\n\n### A guard for libraries that import numpy for you\n\nIf your entry point imports a helper module that imports pandas at module scope, you have already lost. A defensive pattern:\n\n```python\n# sitecustomize.py or a bootstrap module imported first\nimport os\n\nfor var in (\n    \"OPENBLAS_NUM_THREADS\",\n    \"MKL_NUM_THREADS\",\n    \"OMP_NUM_THREADS\",\n    \"NUMEXPR_NUM_THREADS\",\n):\n    os.environ.setdefault(var, \"1\")\n```\n\nDrop this in a `sitecustomize.py` on `PYTHONPATH` and it runs before anything else in the interpreter, including numpy. It is heavy-handed for a laptop, but on a scheduled server it removes an entire class of ordering bugs.\n\n## When Single-Threading Wins, and When It Loses\n\nSingle-threading is not universally faster. It is faster when the machine is oversubscribed or the workload is memory-bound. It is slower when a single job has the box to itself and does real dense linear algebra.\n\n### Wins\n\n- **Overlapping scheduled jobs.** Three jobs each wanting eight threads on eight cores is worse than three jobs each wanting one.\n- **Small matrices.** A 200×200 `dgemm` does not have enough work to amortize thread wake-up. OpenBLAS's own heuristic often decides not to parallelize; forcing one thread removes the decision cost.\n- **Memory-bound operations.** Elementwise ops, `np.where`, `pandas` groupby aggregations — these are limited by RAM bandwidth, not FLOPs. Extra threads add cache-coherence traffic without adding throughput.\n- **Predictable memory.** Each BLAS thread gets its own scratch buffers. Eight threads means eight copies of the workspace. On a box with several jobs, that is the difference between fitting in RAM and swapping.\n- **Container and CI environments.** The container sees the host's core count but may be CPU-limited by cgroups. Threads get throttled and the pool spends time in futex waits.\n\n### Losses\n\n- **A single large dense factorization.** Cholesky on a 5000×5000 matrix, SVD on a big design matrix, a Monte Carlo sweep that is pure `dgemm`. Here BLAS threading is genuinely good, and forcing one thread can cost you 5–10×.\n- **A job that owns the machine.** If nothing else is scheduled and the box has 16 idle cores, pinning to one thread leaves them idle.\n\nThe practical rule: if jobs overlap, or if the workload is not dominated by large dense linear algebra, go single-threaded. If you have one job that does heavy `dgemm` and it runs alone, give it threads — but give it *only* that job, and set the thread count explicitly rather than letting it grab every core.\n\nYou can also split the difference per job. A feature builder that mostly does pandas joins gets `OMP_NUM_THREADS=1`. A nightly calibration that inverts a big matrix gets `OPENBLAS_NUM_THREADS=4` and is scheduled in a window where nothing else runs.\n\n## A Benchmark Harness\n\nDo not guess. Measure on the actual box with the actual workload. Here is a small harness that runs a representative operation at several thread counts and reports wall time and peak RSS.\n\n```python\n\"\"\"bench_threads.py — compare BLAS thread counts on a real workload.\"\"\"\nimport os\nimport subprocess\nimport sys\nimport time\nimport json\n\n# The child process does the actual work so the environment is clean.\nCHILD = r\"\"\"\nimport os, time, json, sys\nimport numpy as np\n\nn = int(sys.argv[1])\nreps = int(sys.argv[2])\nrng = np.random.default_rng(0)\n\na = rng.standard_normal((n, n))\nb = rng.standard_normal((n, n))\n\n# warm up\n_ = a @ b\n\nt0 = time.perf_counter()\nfor _ in range(reps):\n    c = a @ b\nt1 = time.perf_counter()\n\n# a memory-bound op for contrast\nt2 = time.perf_counter()\nfor _ in range(reps):\n    d = np.sqrt(np.abs(a))\nt3 = time.perf_counter()\n\nprint(json.dumps({\n    \"matmul_s\": (t1 - t0) / reps,\n    \"elementwise_s\": (t3 - t2) / reps,\n}))\n\"\"\"\n\n\ndef run(n: int, reps: int, threads: int) -\u003e dict:\n    env = os.environ.copy()\n    for var in (\"OPENBLAS_NUM_THREADS\", \"MKL_NUM_THREADS\",\n                \"OMP_NUM_THREADS\", \"NUMEXPR_NUM_THREADS\"):\n        env[var] = str(threads)\n\n    proc = subprocess.run(\n        [sys.executable, \"-c\", CHILD, str(n), str(reps)],\n        env=env, capture_output=True, text=True, check=True,\n    )\n    return json.loads(proc.stdout.strip().splitlines()[-1])\n\n\nif __name__ == \"__main__\":\n    n = 2000\n    reps = 3\n    for threads in (1, 2, 4, 8):\n        r = run(n, reps, threads)\n        print(f\"threads={threads:\u003e2}  \"\n              f\"matmul={r['matmul_s']*1000:8.1f} ms  \"\n              f\"elementwise={r['elementwise_s']*1000:8.1f} ms\")\n```\n\nRun it on the target machine. On a typical 8-core VM with a 2000×2000 matmul you will see `matmul` drop steadily from 1 to 8 threads, while `elementwise` barely moves — it is memory-bound and the extra threads do nothing. That is the shape of the trade-off.\n\nNow run it *twice concurrently* to simulate overlap:\n\n```python\nimport concurrent.futures as cf\n\nwith cf.ThreadPoolExecutor(max_workers=2) as pool:\n    futures = [pool.submit(run, 2000, 3, t) for t in (1, 8)]\n    for f in futures:\n        print(f.result())\n```\n\nWith `threads=8` on both, the combined time is usually worse than running them sequentially. With `threads=1` on both, they finish in roughly the time of one job each, in parallel. That is the whole argument for pinning to one thread in a scheduled environment.\n\n### Reading the results\n\nTwo numbers matter: wall time and peak RSS. Wall time tells you whether you gained throughput. Peak RSS tells you whether you gained predictability. In my experience the memory win is the one that stops the 3 a.m. pages — a job that used to balloon to 2 GB under thread contention now sits flat at 600 MB.\n\nIf you want peak RSS in the harness, wrap the child with `psutil`:\n\n```python\nimport psutil, subprocess\n\np = subprocess.Popen([sys.executable, \"-c\", CHILD, \"2000\", \"3\"], env=env)\nproc = psutil.Process(p.pid)\npeak = 0\nwhile p.poll() is None:\n    try:\n        peak = max(peak, proc.memory_info().rss)\n    except psutil.NoSuchProcess:\n        break\n    time.sleep(0.05)\nprint(f\"peak RSS: {peak / 1e6:.1f} MB\")\n```\n\n## Checklist\n\nFor every scheduled numerical job on a shared Windows box:\n\n1. Set `OPENBLAS_NUM_THREADS`, `MKL_NUM_THREADS`, `OMP_NUM_THREADS`, and `NUMEXPR_NUM_THREADS` to `1` in the launcher, before `python.exe`.\n2. If you cannot control the launcher, set them at the top of the entry point with `os.environ.setdefault`, before any numerical import — or use a `sitecustomize.py`.\n3. Benchmark on the actual hardware with the actual matrix sizes. Raise the thread count only for a job that owns the machine and does large dense linear algebra.\n4. Watch peak RSS, not just wall time. Flat memory is the real prize.\n\nThe environment variables are not a micro-optimization. On a box running several jobs, they are the difference between a schedule that finishes and one that drifts into the morning.\n\nMore notes like this ship every week on this site.\n\n---\n\n## Daily Picks\n\n\u003e The following pairs are selected from the multi-timeframe trend scanner (Gate.io futures) and are for technical-analysis study only — not investment advice.\n\u003e Data updated: 2026-10-06 12:36:33\n\n### Long\n\n| Pair | Signal | Price | Take Profit | Stop Loss | R/R |\n|------|--------|-------|-------------|-----------|-----|\n| SKYAI |  | $0.0416 | $0.0433 | $0.0406 | 1:1.6 |\n| RE |  | $0.4991 | $0.5181 | $0.4866 | 1:1.5 |\n\n_2 picks selected. Scanner runs every 15 minutes._\n","positive_reactions_count":0,"cover_image":null,"tag_list":["openblas","mkl","ompnumthreads","threadcontention"],"canonical_url":"https://www.hkcode.dpdns.org/en/posts/8.html","reading_time_minutes":8,"user":{"name":"mattleeee","username":"mattleeee","twitter_username":null,"github_username":"mattleeee","user_id":4160029,"website_url":null,"profile_image":"https://media2.dev.to/dynamic/image/width=640,height=640,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png","profile_image_90":"https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png"}},{"type_of":"article","id":4804411,"title":"Fail-Fast Retry Design for Scheduled Scripts: Three Attempts with Backoff","description":"Unattended scripts fail for the least interesting reasons. A DNS lookup times out at 02:00. An API...","published":true,"published_at":"2026-10-06T04:23:43.827Z","slug":"fail-fast-retry-design-for-scheduled-scripts-three-attempts-with-backoff-879","path":"/mattleeee/fail-fast-retry-design-for-scheduled-scripts-three-attempts-with-backoff-879","url":"https://dev.to/mattleeee/fail-fast-retry-design-for-scheduled-scripts-three-attempts-with-backoff-879","comments_count":0,"public_reactions_count":0,"page_views_count":72,"published_timestamp":"2026-10-06T04:23:43Z","body_markdown":"Unattended scripts fail for the least interesting reasons. A DNS lookup times out at 02:00. An API returns 429 because a cron job on another box happened to fire at the same second. A TCP connection resets halfway through a response. None of these are bugs in your logic, and all of them will page you if the job is important enough.\n\nThe instinct is to wrap everything in a `try/except` and loop until it works. That instinct produces a worse failure mode: jobs that hang forever, hide real errors, and leave the scheduler unable to tell whether the previous run finished. This post describes a retry wrapper that stays small, fails fast, and gives the Windows Task Scheduler layer enough information to act.\n\n## The failure modes worth designing for\n\nBefore writing the decorator, it helps to be explicit about what counts as retryable. In practice, the set is narrow:\n\n- Network-level errors: connection reset, timeout, DNS failure.\n- HTTP 5xx responses.\n- HTTP 429 with a `Retry-After` header or a known rate limit.\n- Transient file locks on a shared drive (common on Windows when another process holds a handle).\n\nEverything else should fail immediately. A 401 means your token expired — retrying three times just wastes thirty seconds before the same error. A 400 means your payload is malformed — retrying will never fix it. A `KeyError` in your parsing code means the upstream schema changed, and no amount of backoff will help.\n\nThe practical rule: retry on exceptions that represent *transport* problems, not *logic* problems. In Python, that usually means `requests.RequestException` and its subclasses, plus a small allowlist of OS-level errors.\n\n## A decorator that does one thing\n\nHere is the core implementation. It retries up to three times with linear backoff — 10, 20, then 30 seconds — and logs each attempt with its failure reason.\n\n```python\nimport functools\nimport logging\nimport time\n\nimport requests\n\nlogger = logging.getLogger(__name__)\n\nRETRYABLE = (requests.RequestException,)\n\n\ndef retry(max_attempts: int = 3, base_delay: int = 10):\n    \"\"\"Retry a function on transport errors with linear backoff.\n\n    Delays are base_delay * attempt: 10s, 20s, 30s for the default.\n    KeyboardInterrupt and SystemExit propagate untouched.\n    \"\"\"\n    def decorator(func):\n        @functools.wraps(func)\n        def wrapper(*args, **kwargs):\n            last_exc = None\n            for attempt in range(1, max_attempts + 1):\n                try:\n                    return func(*args, **kwargs)\n                except RETRYABLE as exc:\n                    last_exc = exc\n                    logger.warning(\n                        \"%s attempt %d/%d failed: %s: %s\",\n                        func.__name__, attempt, max_attempts,\n                        type(exc).__name__, exc,\n                    )\n                    if attempt \u003c max_attempts:\n                        delay = base_delay * attempt\n                        logger.info(\"Sleeping %ds before retry\", delay)\n                        time.sleep(delay)\n            logger.error(\n                \"%s exhausted %d attempts, last error: %s\",\n                func.__name__, max_attempts, last_exc,\n            )\n            raise last_exc\n        return wrapper\n    return decorator\n```\n\nThree details matter here.\n\n**`except RETRYABLE`, not `except Exception`.** Catching `Exception` would swallow `KeyboardInterrupt`? No — `KeyboardInterrupt` inherits from `BaseException`, not `Exception`, so a bare `except Exception` would actually let it through. But `except Exception` *would* catch `SystemExit`? Also no. The real danger is catching `AttributeError`, `KeyError`, `ValueError` — logic errors that retrying cannot fix. Those should crash the job immediately so you see them in the log at 02:01, not at 02:06 after three pointless sleeps.\n\n**Explicit re-raise after the loop.** If you let the loop fall through, the function returns `None`, and the caller has no idea anything went wrong. Re-raising `last_exc` preserves the original traceback context and lets the exit-code layer above decide what to do.\n\n**Logging on every attempt, not just the final failure.** When you're debugging a flaky 3 AM job, the pattern matters. Three timeouts in a row points at the network. A 429 followed by two successes points at rate limiting. A 500 followed by a 200 followed by a 500 points at a load balancer routing to a bad backend. You only see these patterns if you log each attempt.\n\n## Backoff shape: linear vs exponential\n\nLinear backoff (10, 20, 30) is the right default for scheduled jobs that run on a fixed cadence. If your job runs every five minutes, the total retry window of 60 seconds fits comfortably inside the interval. Exponential backoff (10, 20, 40, 80…) is better for interactive or high-frequency callers where you want to shed load quickly, but for a job that fires every N minutes, linear keeps the worst-case runtime predictable.\n\nIf you need to honor a server-provided delay, extract it before sleeping:\n\n```python\ndef _retry_after(exc: requests.RequestException) -\u003e int | None:\n    resp = getattr(exc, \"response\", None)\n    if resp is None or resp.status_code != 429:\n        return None\n    header = resp.headers.get(\"Retry-After\")\n    if header and header.isdigit():\n        return int(header)\n    return None\n```\n\nThen in the loop, prefer `_retry_after(exc)` over the computed delay when it's present. This is the one case where the server knows better than you do.\n\n## Exit codes: the contract with the scheduler\n\nWindows Task Scheduler records the exit code of every run, and you can filter on it in the history view. More importantly, a wrapper script can branch on it. The useful convention is small and unambiguous:\n\n| Code | Meaning |\n|------|---------|\n| 0 | Success — work was done |\n| 1 | Failed after retries — investigate |\n| 2 | Nothing to do — no work this cycle |\n| 3 | Configuration or setup error — fix before next run |\n\nThe distinction between 1 and 2 is the one people skip, and it's the one that saves the most time. A job that polls an API for new records will legitimately find nothing most cycles. If \"nothing to do\" returns 1, your monitoring is useless — every quiet night looks like an outage. If it returns 0, you can't tell from the logs whether the job actually ran.\n\nHere's how the wrapper looks:\n\n```python\nimport sys\n\nEXIT_OK = 0\nEXIT_FAILED = 1\nEXIT_NOTHING = 2\nEXIT_CONFIG = 3\n\n\ndef main() -\u003e int:\n    try:\n        records = fetch_pending()\n    except requests.RequestException:\n        return EXIT_FAILED\n    except ConfigError:\n        return EXIT_CONFIG\n\n    if not records:\n        logger.info(\"No pending work\")\n        return EXIT_NOTHING\n\n    process(records)\n    return EXIT_OK\n\n\nif __name__ == \"__main__\":\n    sys.exit(main())\n```\n\nThe `fetch_pending` function is decorated with `@retry()`. When it exhausts its attempts, it re-raises, `main` catches the specific exception type, and returns `EXIT_FAILED`. The scheduler sees 1, your alert rule fires, and the log already contains three timestamped failure reasons.\n\n## Why retry state must live on disk\n\nThis is the part that bites people who test their retry logic interactively and then deploy it. Consider a job that processes a queue of items:\n\n```python\n@retry()\ndef process_batch():\n    items = load_queue()  # reads from disk\n    for item in items:\n        call_api(item)\n        mark_done(item)\n```\n\nThe retry state here is *the queue on disk*. If attempt 1 processes items 1–5 and then fails on item 6, attempt 2 reloads the queue and sees items 6 onward. That's correct. The progress is durable because it lives in a file, not in a variable.\n\nNow consider the in-memory version:\n\n```python\n@retry()\ndef process_batch():\n    items = load_queue()\n    for item in items:\n        call_api(item)\n        mark_done(item)\n    return items  # cached in memory\n```\n\nIf you cache `items` at module level to avoid re-reading, a retry after a partial failure will re-process items 1–5. Depending on the API, that's either wasted work or a duplicate side effect. Worse, if the process is killed between attempts — a reboot, a `taskkill`, a power event — any in-memory state is gone entirely, and the next scheduled run has no idea what happened.\n\nThe rule: **the retry loop should re-read its input from the durable source on every attempt.** The decorator doesn't know or care what the function does; it just calls it again. It's the function's responsibility to start from a consistent, on-disk state each time.\n\nThis has a second benefit. If the job crashes hard — no retry, just a blue screen — the next scheduled run picks up from the same disk state. The retry decorator handles the fast failures; the disk state handles the slow ones.\n\n## Wiring it into Task Scheduler\n\nThe scheduler itself has retry settings, and they interact with yours. Under the task's *Settings* tab, \"If the task fails, restart every\" is a separate mechanism from your in-process retry. Keep them distinct:\n\n- **In-process retry** handles transient errors within a single run. Fast, cheap, no scheduler involvement.\n- **Scheduler restart** handles process-level failures: a crash, a hung process killed by a timeout, a machine that rebooted mid-run.\n\nA reasonable pairing is three in-process attempts with linear backoff, and one scheduler restart after a short delay. That gives you four total chances spread over a few minutes without turning a broken job into an infinite loop.\n\nOne Windows-specific gotcha: if your task uses \"Stop the task if it runs longer than\", set that duration longer than your worst-case retry window. Three attempts with 10/20/30 second backoff plus request timeouts can easily exceed two minutes. If the scheduler kills the process at 60 seconds, you never see the third attempt, and the exit code is a generic timeout rather than your `EXIT_FAILED`.\n\n## What this design deliberately doesn't do\n\nIt doesn't retry on HTTP 4xx (except 429). It doesn't use jitter, because scheduled jobs don't have the thundering-herd problem that interactive clients do. It doesn't persist retry counts across runs — if the job fails three times and the scheduler restarts it, that's a fresh three attempts, and the log will show eight failures total, which is a signal in itself.\n\nIt also doesn't try to be clever about partial success. If a batch of 100 items processes 60 and then fails, the retry starts from item 61 because that's what the disk says. If your API doesn't support idempotent writes, you need a different design — one that records the exact operation before calling out, not after.\n\nThe whole wrapper is about forty lines. Most of the value is in the parts that *don't* retry: the narrow exception tuple, the explicit re-raise, the exit code that separates \"failed\" from \"empty.\" Fail fast, log everything, and let the disk be the source of truth.\n\nMore notes like this ship every week on this site.\n\n---\n\n## Daily Picks\n\n\u003e The following pairs are selected from the multi-timeframe trend scanner (Gate.io futures) and are for technical-analysis study only — not investment advice.\n\u003e Data updated: 2026-10-06 12:36:33\n\n### Long\n\n| Pair | Signal | Price | Take Profit | Stop Loss | R/R |\n|------|--------|-------|-------------|-----------|-----|\n| SKYAI |  | $0.0416 | $0.0433 | $0.0406 | 1:1.6 |\n| RE |  | $0.4991 | $0.5181 | $0.4866 | 1:1.5 |\n\n_2 picks selected. Scanner runs every 15 minutes._\n","positive_reactions_count":0,"cover_image":null,"tag_list":["retry","backoff","resilience","python"],"canonical_url":"https://www.hkcode.dpdns.org/en/posts/3.html","reading_time_minutes":7,"user":{"name":"mattleeee","username":"mattleeee","twitter_username":null,"github_username":"mattleeee","user_id":4160029,"website_url":null,"profile_image":"https://media2.dev.to/dynamic/image/width=640,height=640,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png","profile_image_90":"https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png"}},{"type_of":"article","id":4804400,"title":"Running Scheduled Tasks Silently: The VBS Wrapper Pattern","description":"The Problem: Black Windows Every Hour   A monitoring server had nine scheduled tasks, each...","published":true,"published_at":"2026-10-06T04:22:45.603Z","slug":"running-scheduled-tasks-silently-the-vbs-wrapper-pattern-acc","path":"/mattleeee/running-scheduled-tasks-silently-the-vbs-wrapper-pattern-acc","url":"https://dev.to/mattleeee/running-scheduled-tasks-silently-the-vbs-wrapper-pattern-acc","comments_count":0,"public_reactions_count":0,"page_views_count":80,"published_timestamp":"2026-10-06T04:22:45Z","body_markdown":"## The Problem: Black Windows Every Hour\n\nA monitoring server had nine scheduled tasks, each firing hourly. The machine stays logged in permanently—it runs a dashboard that needs an interactive session. That detail matters, because it meant every task launched a console window on the desktop. Nine tasks, hourly, each flashing a black CMD rectangle over whatever was on screen. Unacceptable for a machine someone actually looks at.\n\nThe tasks did real work: pulling metrics, rotating logs, syncing a couple of directories. They were created over time by different tools—some by hand with `schtasks`, some by a third-party installer, one by an old deployment script. That mix turned out to be the source of the second problem.\n\n## Diagnosis: Why `schtasks /change` Failed\n\nThe obvious fix is to flip each task to run whether the user is logged on or not, which suppresses the window:\n\n```batchfile\nschtasks /change /tn \"\\Metrics\\HourlyPull\" /ru \"SYSTEM\"\n```\n\nThat failed with a parameter error:\n\n```plaintext\nERROR: The parameter is incorrect.\n```\n\nNo further detail. The task existed, the name was right, the quoting was right. I checked the XML with `schtasks /query /tn \"\\Metrics\\HourlyPull\" /xml` and the task definition looked normal. The failure was not in my command.\n\nThe real cause is in how `schtasks.exe` is built. It is a thin command-line parser sitting on top of the Task Scheduler COM API (the `ITaskService` interface family). When you run `/change`, it reads the existing task, mutates a few properties, and writes it back. Tasks created by other tools—installers, deployment frameworks, anything that talks to the COM API directly—often carry property combinations that the `schtasks` parser does not expect. In this case, the offending task had a principal block with a logon type the CLI did not know how to round-trip. The COM call inside returned `E_INVALIDARG`, and `schtasks.exe` surfaced it as the generic \"parameter is incorrect.\"\n\nThe lesson: when a task was not created by `schtasks`, do not try to fix it with `schtasks`. Go to the same API layer the creating tool used.\n\n### PowerShell Reaches the Same API Without the Parser\n\n`Set-ScheduledTask` and `Get-ScheduledTask` are PowerShell cmdlets that wrap the identical COM interface. They do not re-parse a command line, so they do not choke on properties the CLI does not understand. The same mutation that failed above worked immediately:\n\n```powershell\n$t = Get-ScheduledTask -TaskName \"HourlyPull\" -TaskPath \"\\Metrics\\\"\n$t.Principal.LogonType = \"S4U\"\n$t.Principal.RunLevel = \"Highest\"\nSet-ScheduledTask -InputObject $t\n```\n\n`S4U` (service-for-user) runs the task without storing a password and without an interactive session, which is exactly the behavior needed to suppress the console window. `Highest` avoids the elevation prompt path.\n\nBut this only solved part of the problem. Switching every task to `S4U` changes the security context, and a couple of the tasks depended on the interactive user's mapped drives and environment. Rewriting all nine tasks was more invasive than necessary. There is a lighter, more surgical pattern.\n\n## The VBS Wrapper Pattern\n\nThe window appears because the task action is `cmd.exe` or `powershell.exe`, and those are console applications. When Task Scheduler launches a console application in an interactive session, Windows creates a console window. You can suppress that at the launch site rather than by changing the task principal.\n\nThe launch site is `wscript.exe`. It is a GUI-subsystem host: it has no console of its own. When it starts a child process, it can ask the shell to run that child with a hidden window. That is the entire trick.\n\n### The Two-Line Runner\n\nCreate `silent_runner.vbs`:\n\n```vbscript\nSet sh = CreateObject(\"WScript.Shell\")\nsh.Run \"cmd /c \"\"\" \u0026 WScript.Arguments(0) \u0026 \"\"\"\", 0, False\n```\n\nLine by line:\n\n- `CreateObject(\"WScript.Shell\")` gives us the shell automation object.\n- `sh.Run` takes three arguments: the command line, the window style, and whether to wait.\n- Window style `0` means hidden. This is the parameter that kills the black rectangle.\n- `False` means do not wait for the child to exit. The task returns immediately; the actual work runs detached.\n\nThe command line is `cmd /c \"target.bat\"`. The double quotes inside the VBS string are escaped by doubling them, so the final command is `cmd /c \"\u003cpath to bat\u003e\"`. The `cmd /c` wrapper is there so the batch file runs in a proper command interpreter with its own environment, and exits when done.\n\nTwo lines, no dependencies, works on every Windows version since XP.\n\n### Wiring It Into Task Scheduler\n\nEach task action changes from:\n\n```plaintext\nProgram:   C:\\Scripts\\metrics_pull.bat\nArguments: (none)\n```\n\nto:\n\n```plaintext\nProgram:   wscript.exe\nArguments: \"C:\\Scripts\\silent_runner.vbs\" \"C:\\Scripts\\metrics_pull.bat\"\n```\n\nThat is the whole change. The task still runs as the same user, in the same session, with the same environment. Only the window is gone.\n\n### The PowerShell Loop\n\nNine tasks, done in one pass. This script rewrites the action of every task under a given path:\n\n```powershell\n$runner = \"C:\\Scripts\\silent_runner.vbs\"\n$path   = \"\\Metrics\\\"\n\nGet-ScheduledTask -TaskPath $path | ForEach-Object {\n    $task = $_\n    foreach ($action in $task.Actions) {\n        # Only rewrite actions that launch a console host directly.\n        if ($action.Execute -match '^(cmd|powershell|pwsh)(\\.exe)?$') {\n            $target = $action.Arguments\n            if ($target -match '^/c\\s+(.+)$') { $target = $matches[1].Trim('\"') }\n\n            $action.Execute   = \"wscript.exe\"\n            $action.Arguments = \"`\"$runner`\" `\"$target`\"\"\n        }\n    }\n    Set-ScheduledTask -InputObject $task | Out-Null\n    Write-Host \"Rewrote $($task.TaskName)\"\n}\n```\n\nA few things worth noting:\n\n- `Get-ScheduledTask -TaskPath` returns the task objects, and `Set-ScheduledTask -InputObject` writes them back through the COM API. No CLI parsing, no `E_INVALIDARG`.\n- The `Execute` check is a guard. If a task already runs a non-console program, leave it alone.\n- The regex `^/c\\s+(.+)$` strips a leading `/c` from the arguments, which is how `schtasks` stores a `cmd /c` invocation. If the arguments are empty, `$target` stays empty and the rewrite is skipped by the guard—handle that case if your tasks vary.\n- `Out-Null` keeps the pipeline quiet; `Set-ScheduledTask` returns the updated task object otherwise.\n\nIf you want to be more conservative, add a `-WhatIf` style dry run first by printing `$action.Execute` and `$action.Arguments` before mutating.\n\n## Verification\n\nTwo checks, both fast.\n\nFirst, confirm the task definition changed:\n\n```powershell\n(Get-ScheduledTask -TaskName \"HourlyPull\" -TaskPath \"\\Metrics\\\").Actions |\n    Format-List Execute, Arguments\n```\n\nExpected output:\n\n```plaintext\nExecute   : wscript.exe\nArguments : \"C:\\Scripts\\silent_runner.vbs\" \"C:\\Scripts\\metrics_pull.bat\"\n```\n\nSecond, confirm the behavior. Run the task on demand and watch the desktop:\n\n```powershell\nStart-ScheduledTask -TaskName \"HourlyPull\" -TaskPath \"\\Metrics\\\"\n```\n\nNo window should appear. Then check that the work actually happened—the log file, the output directory, whatever the batch file produces. A hidden window that fails silently is worse than a visible one, so verify the side effects, not just the absence of the window.\n\nIf the task still flashes a window, the usual cause is that `wscript.exe` is not the actual launcher. Some tasks are configured with an \"action\" that is a script host already, or a task created by a tool that wraps the action in its own launcher. Query the raw XML to see what is really stored:\n\n```powershell\nExport-ScheduledTask -TaskName \"HourlyPull\" -TaskPath \"\\Metrics\\\"\n```\n\nLook at the `\u003cExec\u003e` block. If `\u003cCommand\u003e` is not `wscript.exe`, the rewrite did not take.\n\n## When This Pattern Is the Right Choice\n\nThe VBS wrapper is not the only way to hide a task window. The alternatives and their trade-offs:\n\n- **Run as `S4U` or `SYSTEM`**: suppresses the window because there is no interactive session, but changes the security context. Breaks tasks that depend on mapped drives, user environment variables, or interactive credentials.\n- **Use `pythonw.exe` instead of `python.exe`**: works if your task is a Python script, but not if it is a batch file or a PowerShell script. Also changes how stdout and stderr behave.\n- **Set the task to \"Run whether user is logged on or not\"**: same security-context change as `S4U`.\n- **The VBS wrapper**: keeps the same user, same session, same environment, same stdout/stderr behavior. Only the window is hidden. This is the least invasive option, which is why it was the right one here.\n\nThe pattern generalizes. Any time you need to launch a console process without a console window—from Task Scheduler, from a startup script, from another automation tool—`wscript.exe` plus a two-line VBS is the smallest possible shim. It has no runtime dependency, no installation, and it survives Windows updates because it uses an API that has been stable for two decades.\n\n## Adapting It to Python\n\nIf you prefer to keep everything in Python, you can generate the VBS and the task definitions from a script. This is useful when you have many tasks to convert or when the task list changes:\n\n```python\nimport subprocess\nfrom pathlib import Path\n\nRUNNER = Path(r\"C:\\Scripts\\silent_runner.vbs\")\nRUNNER.write_text(\n    'Set sh = CreateObject(\"WScript.Shell\")\\n'\n    'sh.Run \"cmd /c \"\"\" \u0026 WScript.Arguments(0) \u0026 \"\"\"\", 0, False\\n',\n    encoding=\"ascii\",\n)\n\ndef rewrite_task(task_name: str, task_path: str, target_bat: str) -\u003e None:\n    \"\"\"Point an existing task at the silent runner.\"\"\"\n    ps = (\n        f\"$a = (Get-ScheduledTask -TaskName '{task_name}' \"\n        f\"-TaskPath '{task_path}').Actions[0]; \"\n        f\"$a.Execute = 'wscript.exe'; \"\n        f\"$a.Arguments = '\\\"{RUNNER}\\\" \\\"{target_bat}\\\"'; \"\n        f\"Set-ScheduledTask -TaskName '{task_name}' -TaskPath '{task_path}' \"\n        f\"-Action $a | Out-Null\"\n    )\n    subprocess.run([\"powershell\", \"-NoProfile\", \"-Command\", ps], check=True)\n\nfor name in (\"HourlyPull\", \"HourlyRotate\", \"HourlySync\"):\n    rewrite_task(name, r\"\\Metrics\\\\\", rf\"C:\\Scripts\\{name}.bat\")\n```\n\nThe Python layer is just orchestration. The actual suppression still happens in the VBS, because that is the only place the window style can be set.\n\n## Summary\n\nThe failure mode was specific: `schtasks /change` returned a generic parameter error on tasks created by other tools, because the CLI's parser could not round-trip properties the COM API produced. The fix was to stop using the CLI and use PowerShell's `Set-ScheduledTask`, which reaches the same API without the parsing layer. The window suppression was a separate concern, solved by routing each task action through `wscript.exe` and a two-line VBS that runs the real command with window style `0`. Nine tasks, one loop, no more black rectangles.\n\nMore notes like this ship every week on this site.\n\n---\n\n## Daily Picks\n\n\u003e The following pairs are selected from the multi-timeframe trend scanner (Gate.io futures) and are for technical-analysis study only — not investment advice.\n\u003e Data updated: 2026-10-06 12:36:33\n\n### Long\n\n| Pair | Signal | Price | Take Profit | Stop Loss | R/R |\n|------|--------|-------|-------------|-----------|-----|\n| SKYAI |  | $0.0416 | $0.0433 | $0.0406 | 1:1.6 |\n| RE |  | $0.4991 | $0.5181 | $0.4866 | 1:1.5 |\n\n_2 picks selected. Scanner runs every 15 minutes._\n","positive_reactions_count":0,"cover_image":null,"tag_list":["wscript","vbs","silenttask","taskscheduler"],"canonical_url":"https://www.hkcode.dpdns.org/en/posts/2.html","reading_time_minutes":7,"user":{"name":"mattleeee","username":"mattleeee","twitter_username":null,"github_username":"mattleeee","user_id":4160029,"website_url":null,"profile_image":"https://media2.dev.to/dynamic/image/width=640,height=640,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png","profile_image_90":"https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png"}},{"type_of":"article","id":4804362,"title":"Why I Replaced Daemon Processes with Windows Task Scheduler","description":"Corporate Windows machines are hostile territory for long-running processes. You write a daemon, it...","published":true,"published_at":"2026-10-06T04:15:40.732Z","slug":"why-i-replaced-daemon-processes-with-windows-task-scheduler-bpe","path":"/mattleeee/why-i-replaced-daemon-processes-with-windows-task-scheduler-bpe","url":"https://dev.to/mattleeee/why-i-replaced-daemon-processes-with-windows-task-scheduler-bpe","comments_count":0,"public_reactions_count":0,"page_views_count":26,"published_timestamp":"2026-10-06T04:15:40Z","body_markdown":"Corporate Windows machines are hostile territory for long-running processes. You write a daemon, it runs fine on your dev box, then you deploy it to a managed laptop and it dies at some random point in the night with no traceback and no Windows Event Log entry. I spent a few weeks chasing this before giving up on the daemon pattern entirely and moving everything to Windows Task Scheduler. Here's what happened and how the migration looks in practice.\n\n## The Symptom: A Daemon That Dies Quietly\n\nThe first version of my job runner was a classic Python daemon: a parent process that spawned child workers, a watchdog thread that restarted workers if they crashed, and a log file that recorded every restart. On my own machine it ran for weeks.\n\nOn the corporate laptop, the log looked like this:\n\n```plaintext\n2024-03-11 09:14:02 INFO  worker started (pid 18432)\n2024-03-11 09:14:02 INFO  worker pid 18432 exited with code 0\n2024-03-11 09:14:02 INFO  restarting worker...\n2024-03-11 09:14:02 INFO  worker started (pid 19004)\n2024-03-11 09:14:02 INFO  worker pid 19004 exited with code 0\n2024-03-11 09:14:02 INFO  restarting worker...\n```\n\nA tight restart loop with no error. The child processes were being created and immediately killed. No Python exception, because the kill happened at the OS level before the interpreter could raise anything.\n\nThe culprit was Sangfor aTrust, the endpoint security suite that ships with a lot of enterprise environments. It hooks process creation and, depending on policy, blocks or silently terminates child processes that don't match an allowlist. A daemon that spawns workers looks exactly like the kind of behavior a security product is built to stop. The watchdog made it worse: every restart attempt was another blocked `CreateProcess` call, which is why the loop was so tight.\n\nYou can confirm this pattern quickly. Run a trivial spawner and see whether the child ever gets to execute:\n\n```python\nimport subprocess\nimport sys\n\n# If this prints nothing, the child never ran.\n# That is a security-policy block, not a Python bug.\nproc = subprocess.Popen(\n    [sys.executable, \"-c\", \"print('child alive')\"],\n    stdout=subprocess.PIPE,\n    stderr=subprocess.PIPE,\n)\nout, err = proc.communicate(timeout=10)\nprint(\"returncode:\", proc.returncode)\nprint(\"stdout:\", out)\nprint(\"stderr:\", err)\n```\n\nIf `returncode` is non-zero and both streams are empty, the child was killed before it could write anything. That's the signature.\n\n## Why Task Scheduler Survives Where Daemons Don't\n\nThe key insight is that Task Scheduler is not a user process. It's a Windows service (`Schedule`) running in session 0, and the processes it launches are created by the service host, not by your Python interpreter. Sangfor aTrust and similar suites generally allow this path because:\n\n1. **The parent is a trusted system service.** Process-creation hooks are usually scoped by parent identity. `svchost.exe` running the Schedule service is on the allowlist; `python.exe` spawning `python.exe` is not.\n2. **Task Scheduler is a documented management surface.** Enterprise security products are designed not to break the OS's own scheduling. Blocking it would break Windows Update, backup agents, and half the management stack. So it stays open.\n3. **No persistent parent process.** There's nothing long-running to kill. Each task run is a short-lived process, launched, executed, and reaped by the service. The \"daemon\" is the scheduler itself, which you don't have to keep alive.\n\nThat last point is the real architectural win. Instead of building a process that stays up and supervises itself, you hand the supervision to the OS. The OS is very good at it and the security suite leaves it alone.\n\n## Mapping Jobs to Tasks\n\nOnce I accepted the model, the migration was mechanical. Each recurring job becomes one scheduled task with four things defined: a trigger, a working directory, a timeout, and a result convention.\n\n### Trigger\n\nTriggers replace your `while True: sleep(...)` loop. For a job that runs every five minutes, use a daily trigger with a repetition interval. For a job that runs once at 06:00 on weekdays, use a weekly trigger with the day set. The scheduler handles the clock; you don't need `time.sleep` at all.\n\n### Working Directory\n\nThis is the most common source of \"it works in my terminal but not in the task.\" Task Scheduler defaults the working directory to `C:\\Windows\\System32`. Any relative path in your script breaks. Always set an explicit working directory, and make your script resolve its own paths rather than relying on `os.getcwd()`:\n\n```python\nfrom pathlib import Path\n\n# Anchor everything to the script location, not the process cwd.\nBASE_DIR = Path(__file__).resolve().parent\nLOG_DIR = BASE_DIR / \"logs\"\nSTATE_DIR = BASE_DIR / \"state\"\nLOG_DIR.mkdir(exist_ok=True)\nSTATE_DIR.mkdir(exist_ok=True)\n```\n\n### Timeout\n\nTask Scheduler has a per-task execution time limit. Set it. A hung job that holds a database connection or a lock file forever is worse than a failed job. For a job expected to run in under two minutes, set the limit to ten. The scheduler terminates the process and records a failure, which you can see.\n\n### Result Convention\n\nThe scheduler records the process exit code. Use it. A script that always exits 0 is invisible to the scheduler's history. Exit non-zero on failure so the last-run result is meaningful:\n\n```python\nimport sys\nimport logging\n\nlogging.basicConfig(\n    filename=str(LOG_DIR / \"job.log\"),\n    level=logging.INFO,\n    format=\"%(asctime)s %(levelname)s %(message)s\",\n)\n\ndef main() -\u003e int:\n    try:\n        run_job()\n    except Exception:\n        logging.exception(\"job failed\")\n        return 1\n    return 0\n\nif __name__ == \"__main__\":\n    sys.exit(main())\n```\n\n## Creating Tasks from Python\n\nYou can register tasks by hand in the GUI, but that doesn't scale and it isn't reviewable. I keep a single `setup_tasks.py` that registers every job idempotently. It shells out to `schtasks.exe`, which is present on every Windows install and needs no extra dependency.\n\n```python\nimport subprocess\nfrom pathlib import Path\n\nPYTHON = r\"C:\\Python311\\python.exe\"\nBASE = Path(r\"C:\\jobs\")\n\nTASKS = [\n    {\n        \"name\": \"QuantJob_Fetch\",\n        \"script\": BASE / \"fetch.py\",\n        \"schedule\": \"MINUTE /MO 5\",\n        \"timeout_min\": 10,\n    },\n    {\n        \"name\": \"QuantJob_Report\",\n        \"script\": BASE / \"report.py\",\n        \"schedule\": \"DAILY /ST 06:00\",\n        \"timeout_min\": 30,\n    },\n]\n\ndef register(task: dict) -\u003e None:\n    cmd = [\n        \"schtasks\", \"/Create\",\n        \"/TN\", task[\"name\"],\n        \"/TR\", f'\"{PYTHON}\" \"{task[\"script\"]}\"',\n        \"/SC\", *task[\"schedule\"].split(),\n        \"/RL\", \"LIMITED\",          # run with normal user rights\n        \"/F\",                       # overwrite if it exists\n    ]\n    subprocess.run(cmd, check=True, capture_output=True, text=True)\n\n    # Set working directory and execution time limit separately.\n    # schtasks has no direct flag for cwd, so use XML for full control.\n    set_task_xml(task)\n\ndef set_task_xml(task: dict) -\u003e None:\n    # Export, patch, re-import. This is the reliable way to set\n    # WorkingDirectory and ExecutionTimeLimit.\n    name = task[\"name\"]\n    xml = subprocess.run(\n        [\"schtasks\", \"/Query\", \"/TN\", name, \"/XML\"],\n        check=True, capture_output=True, text=True,\n    ).stdout\n\n    xml = xml.replace(\n        \"\u003cWorkingDirectory\u003e\u003c/WorkingDirectory\u003e\",\n        f\"\u003cWorkingDirectory\u003e{BASE}\u003c/WorkingDirectory\u003e\",\n    )\n    xml = xml.replace(\n        \"\u003cExecutionTimeLimit\u003ePT72H\u003c/ExecutionTimeLimit\u003e\",\n        f\"\u003cExecutionTimeLimit\u003ePT{task['timeout_min']}M\u003c/ExecutionTimeLimit\u003e\",\n    )\n    # StartWhenAvailable: if the machine was off at trigger time,\n    # run the task as soon as it comes back.\n    if \"\u003cStartWhenAvailable\u003efalse\u003c/StartWhenAvailable\u003e\" in xml:\n        xml = xml.replace(\n            \"\u003cStartWhenAvailable\u003efalse\u003c/StartWhenAvailable\u003e\",\n            \"\u003cStartWhenAvailable\u003etrue\u003c/StartWhenAvailable\u003e\",\n        )\n\n    subprocess.run(\n        [\"schtasks\", \"/Create\", \"/TN\", name, \"/XML\", \"-\", \"/F\"],\n        input=xml, check=True, capture_output=True, text=True,\n    )\n\nif __name__ == \"__main__\":\n    for t in TASKS:\n        register(t)\n        print(f\"registered {t['name']}\")\n```\n\nThe XML round-trip is the part people skip and then wonder why their working directory is `System32`. `schtasks` doesn't expose `/WorkingDirectory` or `/ExecutionTimeLimit` as command-line flags, so you export the task, edit the XML, and re-import. It's ugly but it's stable and it's scriptable.\n\n## Operational Wins You Get for Free\n\nThe migration wasn't just about surviving the security suite. Three things got better immediately.\n\n### Reboot survival\n\nA daemon dies on reboot and you need a service wrapper, a startup shortcut, or a login script to bring it back. A scheduled task just runs on its trigger. If the trigger is a repetition interval, the task resumes on the next interval after boot. If you need it to catch up, `\u003cStartWhenAvailable\u003etrue\u003c/StartWhenAvailable\u003e` makes the scheduler run a missed task as soon as the machine is back. That single flag replaced an entire startup-recovery code path in my old daemon.\n\n### Built-in retry\n\nTask Scheduler has a native retry policy. If a task fails, you can configure it to retry every N minutes for M attempts. That's the watchdog pattern, implemented by the OS, and it doesn't spawn child processes from your interpreter so the security suite has nothing to object to. You set it in the task's Settings tab, or in XML:\n\n```xml\n\u003cRestartOnFailure\u003e\n  \u003cInterval\u003ePT5M\u003c/Interval\u003e\n  \u003cCount\u003e3\u003c/Count\u003e\n\u003c/RestartOnFailure\u003e\n```\n\n### Last-run result codes\n\nEvery task records its last run time and result code. You can query them without any custom instrumentation:\n\n```python\nimport subprocess\nimport csv\nfrom io import StringIO\n\ndef last_results() -\u003e list[dict]:\n    out = subprocess.run(\n        [\"schtasks\", \"/Query\", \"/FO\", \"CSV\", \"/V\"],\n        check=True, capture_output=True, text=True,\n    ).stdout\n    rows = list(csv.DictReader(StringIO(out)))\n    return [\n        {\n            \"name\": r[\"TaskName\"],\n            \"status\": r[\"Status\"],\n            \"last_run\": r[\"Last Run Time\"],\n            \"result\": r[\"Last Result\"],\n        }\n        for r in rows\n        if r[\"TaskName\"].startswith(\"\\\\QuantJob_\")\n    ]\n\nfor row in last_results():\n    print(row)\n```\n\n`Last Result` is the process exit code. `0` means success, anything else means the job failed and you can look at your log file. That's a free health dashboard for every job on the machine. I pipe this into a small status page and I haven't written a single line of process-monitoring code since.\n\n## What I'd Do Differently\n\nTwo things. First, I'd set explicit working directories from day one instead of debugging `FileNotFoundError` for an afternoon. Second, I'd design every job to be idempotent and short-lived. The scheduler model rewards jobs that start, do one thing, and exit. If your job needs to hold state between runs, write it to a file or a database rather than keeping it in memory. The process is gone when it's done; that's the point.\n\nThe daemon pattern isn't wrong in general. It's wrong on a locked-down corporate machine where you don't control the process-creation policy. Task Scheduler is the platform's own answer to \"run this on a schedule,\" and in that environment, using the platform's answer is the shortest path to something that actually stays running.\n\nMore notes like this ship every week on this site.\n\n---\n\n## Daily Picks\n\n\u003e The following pairs are selected from the multi-timeframe trend scanner (Gate.io futures) and are for technical-analysis study only — not investment advice.\n\u003e Data updated: 2026-10-06 12:36:33\n\n### Long\n\n| Pair | Signal | Price | Take Profit | Stop Loss | R/R |\n|------|--------|-------|-------------|-----------|-----|\n| SKYAI |  | $0.0416 | $0.0433 | $0.0406 | 1:1.6 |\n| RE |  | $0.4991 | $0.5181 | $0.4866 | 1:1.5 |\n\n_2 picks selected. Scanner runs every 15 minutes._\n","positive_reactions_count":0,"cover_image":null,"tag_list":["taskscheduler","daemon","python","sangfor"],"canonical_url":"https://www.hkcode.dpdns.org/en/posts/1.html","reading_time_minutes":7,"user":{"name":"mattleeee","username":"mattleeee","twitter_username":null,"github_username":"mattleeee","user_id":4160029,"website_url":null,"profile_image":"https://media2.dev.to/dynamic/image/width=640,height=640,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png","profile_image_90":"https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160029%2F12bf716b-202f-4545-9d17-535a96c8b29d.png"}}]
Enter fullscreen mode Exit fullscreen mode

Top comments (0)